Idan/fga-navigation

Target-driven visual navigation on factor graph attention

An agent is told an object to find and acts from egocentric observations in AI2-THOR. Each grid cell of the observation is scored against the target's word embedding, the recurrent memory and the last action, by the attention layer of Factor Graph Attention (CVPR 2019). The attention-weighted map -- not a pooled vector -- enters the recurrence, because the agent has to know where the target is, not only that it is present.

Trained by advantage actor-critic on the SAVN offline dump: every reachable pose was rendered once and its ResNet18 feature map cached, so no simulator is needed.

Code: github.com/idansc/fga.

Results

Scored on the published fixed set of 3,914 test episodes, each pinning the scene, the target object instance and the start pose. Checkpoints were selected on the 4,086 val episodes and the winner scored once on test, which is what the original's full_eval.py does.

	success	SPL
this checkpoint	0.421	0.167
released A2C weights, same protocol	0.405	0.167
released final weights, same protocol	0.411	0.163
published figure	0.462	0.179

The released weights were re-scored here rather than taken on trust, and they land in the same range: the roughly four-point offset to the published 0.462 applies to them as much as to this model, so it belongs to the measurement rather than to the training. Their model's SPL reproduces to 0.003, which is what says the environment is faithful.

Two things worth knowing if you build on this. Val and test disagree — the run that scored 0.367 on val scored 0.421 on test — so val selection is not a reliable proxy here even though it is the published protocol. And the agents overfit to training scenes: train success reaches 0.906 while held-out sits near 0.42, and more episodes past about one million make that worse rather than better.

Usage

from fga.tasks.navigation import NavigationPolicy

policy = NavigationPolicy.from_pretrained("Idan/fga-navigation")
out = policy(target_embeds=glove, observation=resnet_grid, hidden_state=state)
action = out.action_logits.argmax(-1)

The episodes come from the SAVN offline data:

curl -O https://prior-datasets.s3.us-east-2.amazonaws.com/savn/data.tar.gz
python scripts/run_navigation.py --data_root data \
    --val_episodes data/val_episodes.json --test_episodes data/test_episodes.json \
    --output_dir models/navigation

Citation

Please cite both the attention and the navigation work:

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}

@inproceedings{mayo2021visual,
  title={Visual Navigation with Spatial Attention},
  author={Mayo, Bar and Hazan, Tamir and Tal, Ayellet},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021}
}

The navigation setting, the offline AI2-THOR data and the episode splits are from barmayo/spatial_attention (arXiv:2104.09807).