SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3. Since this is an end-to-end model, the language and visual features are jointly encoded and trained end-to-end from input language and images to output masks and actions. Some errors arise because the model is attending more to the visuals in front of it rather than the language input. E.g. if the robot is standing very close to a cabinet door, or a shelf with a coffee pot, even though the language instruction was to "go to the laser cannon", it will try to Open and Close the cabinet door, or Pickup the coffee pot, respectively, which the model has learnt from the data.
4. Some more sources of error distributed in the model across all evaluation tasks include ambiguity in language, inability to model longe-horizon task instructions like "turn right, go to the breakroom, heat the bowl and place it on the table" - which consists of at least 9 steps, inability to detect masks for some very small objects, or for objects too up close, and failure to distinguish between very similar looking objects (like Laser Cannon and Freeze Ray). These are common issues between the Neural-Symbolic and the VL models.
5. We also did an ablation study by training only the mask prediction head and using the ground truth action predictions. Evaluating this model on the validation dataset yields a MSR of  $71.89\%$ . This simulates the scenario when the perfect actions are predicted to situate the robot in a position to predict a mask for an interactable object, thus evaluating only object detection and visual grounding of the natural language instruction. This result shows the model's difficulties in finding object locations in an end-to-end fashion, requiring the incorporation of better path planning modules that learn from multi-modal inputs.