LXMERT - Visual features don't match original implementation
Author: eladsegal
Pull request: no

## Environment info
- `transformers` version: 4.0.0-rc-1
- Platform: Linux-4.15.0-122-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@eltoto1219

## Information

Model I am using: **unc-nlp/lxmert-gqa-uncased**

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is: 
* [X] **GQA**

## Details

I've tried to reproduce LXMERT results on GQA -
Using the [visual features from the original repo](https://github.com/airsplay/lxmert#gqa), I got an accuracy of 59.29% on the testdev split (which is a bit less than expected, but close enough).
However, when generating the visual features using the [extraction script](https://github.com/huggingface/transformers/blob/master/examples/lxmert/extracting_data.py) in the examples (which uses "unc-nlp/frcnn-vg-finetuned"), the accuracy is only ~33%.
I checked further, and the bounding boxes are also different.

Any idea to what could be the problem?


Comments:
Hi @eladsegal ! 

This is indeed interesting. The FRCNN config should match the exact setting used in the original demo. I will say, however, everything is finicky, but has been tested + should work.

Could you let me know if you:
- modified any of the settings in the config
- provide a link to the script script you are using so I can see the exact changes you have made
- If also possible, plot the bounding boxes on the original prediction so that I can see how the bounding boxes are different (I think that may be pretty telling)

I think I should be able to figure out what may be happening if you could let me know the above!

Thanks for the response, @eltoto1219!
The 33% I got previously was a mistake, not sure what change I made that caused it. 
I started from scratch now, and the results are a lot closer to the original.
I've made the following changes:
- https://github.com/huggingface/transformers/blob/master/examples/lxmert/utils.py#L552 - do this only if the image is from a URL (https://github.com/huggingface/transformers/issues/8333) (improves accuracy on GQA by ~4 points)

And changes so extracting_data.py will work for batch size larger than 1:
- https://github.com/huggingface/transformers/blob/master/examples/lxmert/extracting_data.py#L95 - changed to if len(batch) > 0

- https://github.com/huggingface/transformers/blob/master/examples/lxmert/modeling_frcnn.py#L42 - added .view(-1, 1, 1) to allow broadcasting

For `extracting_data.py` I made a few more changes to be able to extract only for GQA's testdev images (so it will be a lot faster to run).
You can reproduce everything with the code in:
https://github.com/eladsegal/gqa_lxmert
There's also a notebook there showing the different bounding boxes.

For some reason, inference with different batch sizes in the FRCNN model results in different features for the same image.
I got the following accuracies for GQA for the following different batch sizes:
6: 57.15%
2: 57.64%
1: 58.54%
Using features from the original LXMERT repo results in 59.29%.

Of course! I am am real glad you were able to catch the color format error for images downloaded via a url (It would have taken me forever to find something like that)! I will have to fix that as soon as possible + do some more testing. 

I find the different downstream lxmert accuracies on GQA when different batch sizes are used for feature extraction really interesting aswell.  In the original repo, the extraction was set up so that one image went through the faster frcnn at a time. I am thinking something may be getting unordered when using multiple images at once, so I will take a look. I think it would also be worth pointing out that there are multiple releases of visual genome images (from 2014 and 2016). If downloading directly from https://visualgenome.org/ there appears to be quite a few corrupt images. 

On the other hand,  all visual genome images can be downloaded from https://cs.stanford.edu/people/dorarad/gqa/about.html (which I am assuming is the latest version). That may in part be why the accuracy is lower. (GQA train + val split come from visual genome. GQA testdev split comes from the COCO test split) There may also be some very small changes to some extraction hyper-parameters (most likely the NMS threshold for the post processing of bounding boxes) which may have also resulted in slightly different inaccuracies. 

I'll go ahead and extract features across the different versions of visual genome + gqa, and compare element-wise with the features from the original lxmert repo and see if anything is any different, and if so, how different. It should take me a couple of days, but I will get back to you by then!

Thank you, I really appreciate this!

>I think it would also be worth pointing out that there are multiple releases of visual genome images (from 2014 and 2016). If downloading directly from https://visualgenome.org/ there appears to be quite a few corrupt images.
> 
> On the other hand, all visual genome images can be downloaded from https://cs.stanford.edu/people/dorarad/gqa/about.html (which I am assuming is the latest version). That may in part be why the accuracy is lower. (GQA train + val split come from visual genome. GQA testdev split comes from the COCO test split) 

Not sure about this as a possible reason for the lower accuracy, as the images I used were downloaded from GQA's website, and I only did comparison on the testdev split, and no additional training at all.

> There may also be some very small changes to some extraction hyper-parameters (most likely the NMS threshold for the post processing of bounding boxes) which may have also resulted in slightly different inaccuracies.

Sounds like a very reasonable explanation for the different and features and the small accuracy difference!




@eltoto1219 The issue with batch-wise extraction is known:
https://github.com/airsplay/py-bottom-up-attention/issues/3#issuecomment-624240642

Hey @eladsegal !

I stumbled upon that actually on Friday myself too. I rewrote the extraction script to only allow frcnn extraction for one image at a time. 

I think that the discrepancy in accuracy comes from the fact that Hao actually used a caffe-based frcnn pretrained model trained specifically to predict 36 images, while the pytorch one here was pretrained to predict 10-100 images. That and the potential of slightly different NMS thresholds. 

Aslong as the batch-size is 1, the feature quality is most likely the same, however, if we are finetuning with the features from the model used in the aforementioned script, those technically wont be the exact same features used to pretrain lxmert, they should still get the job done if your okay with being ~1% lower than the reported accuracy.


Here is the link to the fixed script:

https://drive.google.com/file/d/1er2axVyGj8eW84QBGrV0dqTmKbxyS8F7/view?usp=sharing


Thank you very much @eltoto1219, this has been extremely helpful!

Why [`examples/lxmert/`](https://github.com/huggingface/transformers/blob/master/examples/lxmert/) no longer exists?

This was an error, it has been put back a few days ago. Sorry for the inconvenience.