SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Screen: <img id=0 class="IconThreeDots" alt="More Icon"></img> <p id=1 class="text" alt="Search your phone and more"> Search your phone and more </p> <p id=2 class="text" alt="M" M </p> <p id=3 class="text" alt="0" 0 </p> <img id=4 class="IconPlay" alt="Play Icon"></img> <p id=5 class="text" alt="Clock"> Clock </p> <p id=6 class="text" alt="YouTube"> YouTube </p> <p id=7 class="text" alt="Photos"> Photos </p> <p id=8 class="text" alt="Gmail"> Gmail </p> <p id=9 class="text" alt="All apps"> All apps </p> <p id=10 class="text" alt="g"> g </p> <p id=11 class="text" alt="0"> 0 </p> <img id=12 class="IconTakePhoto" alt="Camera Icon"></img> <p id=13 class="text" alt="10"> 10 </p> <p id=14 class="text" alt="Calendar"> Calendar </p> <p id=15 class="text" alt="Camera"> Camera </p> <p id=16 class="text" alt="Chrome"> Chrome </p> <p id=17 class="text" alt="Clock"> Clock </p> <p id=18 class="text" alt="0"> 0 </p> <p id=19 class="text" alt="M"> M </p> <p id=20 class="text" alt="B"> B </p> <img id=21 class="IconPerson" alt="Person Icon"></img> <p id=22 class="text" alt="Gmail"> Gmail </p> <p id=23 class="text" alt="Drive"> Drive </p> <p id=24 class="text" alt="Files"> Files </p> <p id=25 class="text" alt="Contacts"> Contacts </p> <p id=26 class="text" alt="G 00"> G 00 </p> <img id=27 class="IconGoogle" alt="Google Icon"></img> <img id=28 class="IconLocation" alt="Location Icon"></img> <img id=29 class="IconCall" alt="Phone Icon"></img> <img id=30 class="IconChat" alt="Chat Icon"></img> <p id=31 class="text" alt="Google"> Google </p> <p id=32 class="text" alt="Maps"> Maps </p>
Instruction: Search for hotels in Chicago.  
Answer: Let's think step by step. I see the app drawer, I need to search, so the action is {"action_type": "click", "idx": 27}
Previous Actions: <HISTORY>   
Screen: <SCREEN REPRESENTATION>   
Instruction: <GROUNDING_GOAL>   
Answer: Let's think step by step. I see
Appendix C Dataset format
Each datapoint is stored as a TFRecord file with compression type 'GZIP' with the following fields:
- android_api_level: the Android API level of the emulator the episode was collected from
- currentActivity: the name of the activity running when the example was collected
- device_type: the device type of the emulator the episode was collected from, mostly Pixel devices with one custom device image
- episode_id: the unique identifier for the episode the example is from
- episode_length: the overall number of steps in the episode
- goal_info: the natural language instruction the episode is demonstrating
- image/channels, image/height, image/width: the number of channels, height, and width of the screenshot
- image/encoded: the encoded screenshot
- image/ui_annotations_positions: a flattened array of coordinates representing the bounding boxes of the UI annotations; the coordinates are in (y, x, height, width) format and the length of this array is 4 * num_elements
- image/uri_annotations_text: the OCR-detected text associated with the UI element
- image/ui_annotations.ui_types: the type of UI element for each annotation, can be an icon or just text
- results/action_type: the type of the predicted action (see 'Action space' for more details)
- results/type_action: if the action is a type then this holds the text string that was typed
- results/yx_touch, results/yx Lift: the (y, x) coordinates for the touch and lift point of a dual point action
- step_id: the example's zero-indexed step number within the episode (i.e. if step_id is 2, then this is the third step of the episode)