Repository: facebookresearch/mmf

Issue:
BUTD training broken after PR #135
## 🐛 Bug

After PR #135, training of the BUTD method for image captioning is now broken.

This issue is that #135 removes the teacher forcing behavior during training. After #135, now BUTD to use beam search decoding (instead of teacher-forcing) during training, causing a lot of shape mismatch, gradient and other errors.

## Command

## To Reproduce

Steps to reproduce the behavior:

```
python tools/run.py \
    --tasks captioning --datasets coco --model butd \
    --config configs/captioning/coco/butd.yml
```

The training crashes due to various errors (such as batch size mismatch).

## Expected behavior

The training should work as expected.

## Environment

You can run the script with:
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.1 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.9.4

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: 
GPU 0: Quadro GP100
GPU 1: Quadro GP100

Nvidia driver version: 410.79
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] block.bootstrap.pytorch==0.1.5
[pip] bootstrap.pytorch==0.0.13
[pip] msgpack-numpy==0.4.1
[pip] murel.bootstrap.pytorch==0.0.0
[pip] numpy==1.17.2
[pip] numpydoc==0.7.0
[pip] torch==1.2.0
[pip] torchfile==0.1.0
[pip] torchtext==0.3.1
[pip] torchvision==0.4.0
[conda] block.bootstrap.pytorch   0.1.5                     <pip>
[conda] bootstrap.pytorch         0.0.13                    <pip>
[conda] cuda90                    1.0                  h6433d27_0    pytorch
[conda] faiss-gpu                 1.4.0           py36_cuda9.0.176_1  [cuda90]  pytorch
[conda] magma-cuda90              2.3.0                         1    pytorch
[conda] mkl                       2018.0.3                      1  
[conda] mkl-fft                   1.0.0                     <pip>
[conda] mkl-include               2018.0.3                      1  
[conda] mkl-service               1.1.2            py36h17a0993_4  
[conda] mkl_fft                   1.0.2           np114py36_intel_0  [intel]  intel
[conda] mkl_random                1.0.1            py36h629b387_0    anaconda
[conda] mkldnn                    0.14.0                        0    mingfeima
[conda] murel.bootstrap.pytorch   0.0.0                     <pip>
[conda] nccl2                     1.0                           0    pytorch
[conda] torch                     1.2.0                     <pip>
[conda] torchfile                 0.1.0                     <pip>
[conda] torchtext                 0.3.1                     <pip>
[conda] torchvision               0.4.0                     <pip>



Discussion:

Resolving patch:
diff --git a/pythia/models/butd.py b/pythia/models/butd.py
--- a/pythia/models/butd.py
+++ b/pythia/models/butd.py
@@ -119,28 +119,37 @@ def forward(self, sample_list):
             dtype=torch.float,
         )
 
-        decoder = registry.get_decoder_class(self.config["inference"]["type"])(self.vocab, self.config)
+        if self.config["inference"]["type"] in ["beam_search", "nucleus_sampling"]:
+            decoder = registry.get_decoder_class(self.config["inference"]["type"])(
+                self.vocab, self.config
+            )
+            sample_list = decoder.init_batch(sample_list)
 
-        sample_list = decoder.init_batch(sample_list)
-        # batch_size = sample_list.get_batch_size()
         batch_size = sample_list.image_feature_0.size(0)
         data, sample_list, timesteps = self.prepare_data(sample_list, batch_size)
         output = None
         batch_size_t = batch_size
         for t in range(timesteps):
             data, batch_size_t = self.get_data_t(t, data, batch_size_t, output)
-            pi_t = data["texts"]
+            if self.config["inference"]["type"] in ["beam_search", "nucleus_sampling"]:
+                pi_t = data["texts"]
+            else:
+                pi_t = data["texts"][:, t].unsqueeze(-1)
             embedding = self.word_embedding(pi_t)
             attention_feature, _ = self.process_feature_embedding(
                 "image", sample_list, embedding[:, 0, :], batch_size_t=batch_size_t
             )
             output = self.classifier(attention_feature)
             # Compute decoding
-            finish, data, batch_size_t = decoder.decode(t, data, output)
-            if finish:
-                break
+            if self.config["inference"]["type"] in ["beam_search", "nucleus_sampling"]:
+                finish, data, batch_size_t = decoder.decode(t, data, output)
+                if finish:
+                    break
+            else:
+                scores[:batch_size_t, t] = output
 
         model_output = {"scores": scores}
-        model_output["captions"] = decoder.get_result()
+        if self.config["inference"]["type"] in ["beam_search", "nucleus_sampling"]:
+            model_output["captions"] = decoder.get_result()
 
         return model_output


Test patch: