Add EfficientLoFTR model
Author: sbucaille
Pull request: yes
Labels: name: New model

name: Vision

# What does this PR do?

As discussed earlier, here is the implementation of EfficientLoFTR with their official weight. 
This PR is an anticipation of the release of other weights like MatchAnything https://github.com/zju3dv/MatchAnything
Fixes #36354


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
 @qubvel 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi @qubvel , this PR is ready for a review 🤗 

Hi @sbucaille! Super cool 🔥  I will review this week!

Hi @qubvel, thanks for the review !
I've addressed some of the comments, some of them still need to be addressed. 
For the data-dependent operations, at first, I wanted to do like SuperGlue, which is indeed using masks. The thing here is that we deal with dense matching, meaning that 1/8th of the pixels are attempted to be matched. Depending on the matching threshold, we have more of less matches. If we keep the dense, we end up carrying around quite large and potentially sparse tensors with a lot of zeroes. This may have negative impact on the computations of scores for example. 
This is why I fell back to something similar to the original implementation which is using indices, because you end up with smaller tensors. Since the model's goal is supposed to be "efficient" I thought that would be the best way 😅 
Anyway, let me try to implement it with masks and I'll benchmark the inference time of the models to have an idea of the difference.

Okay, sounds good. Let's benchmark both versions. We should benefit from compilation if we can make it without graph breaks, and that benefit might be much greater than operating on smaller tensors. In addition, that should unlock torch.export. In the worst case, we can have two different paths for the eager and compiled versions switched by `is_torchdynamo_compiling()`

Hi @qubvel,
I finished addressing the comments you made.
About graph breaks, I implemented the full dense version that allows compilation and you were right, with compilation we have better performance than the original sparse version. Even without the compilation I managed to have better performance although this comes at a small memory cost.
Here are the results : 
```
dense - compile :
{'average_forward_time (s)': 0.07417428731918335, 'peak_memory_usage (MB)': 590.228515625}

dense :
{'average_forward_time (s)': 0.09448522090911865, 'peak_memory_usage (MB)': 977.041015625}

sparse - compile : impossible

sparse :
{'average_forward_time (s)': 0.10868784666061401, 'peak_memory_usage (MB)': 563.5390625}
```

Not sure why the quality test is not passing, `make style` does not change anything in my repository, is this from main ? 🤔 
About the other test, I'll check in the coming days 🤗 

Hi @qubvel ,
Addressed your comments.
I'll convert back the modeling file into using the `#Copied from` mechanism in the coming days.
Also, I'll need to overwrite the `batching_equivalence` test as different keypoint coordinates from batched and single row inputs cause it to fail.
I'll ping you when it is done.

Hi @qubvel,
I converted back the modeling file using `Copied from` statements.
I added the custom `batching_equivalence` test which takes care of checking how many keypoints are different, similarly to what we did in SuperGlue, also added comments about it.
Let me know what you think

Hey @qubvel , small gentle bump here aswell 🤗 

Hey @qubvel , comments addressed, I increased slightly the tolerance for the `batching_equivalence` test

edit: tests not passing do not seem to be related to the PR

Hey @qubvel , just a small bump here.
But by rebasing on the latest change I end up with an error when the script checks for the importability of all auto mappings. After looking around and comparing with what I did previously with SuperGlue I can't figure out what I am missing, any idea ? 🤔 

Re @qubvel , just realized I forgot a couple of your comments, my bad, these are addressed now. I updated the rope code because some meta device tests were not passing, all good now except the auto import which I still can't figure out 😅 

@qubvel bumping eloftr now that lightglue is merged, I fixed all the details and added `auto_docstring` and `can_return_tuple` decorators as well as cleaned the attention forward to follow the recent changes 

Thanks @sbucaille, I will do a review this week! Sorry for the delay 😅 

These are nice QoL changes, I'll keep them in mind for the next PRs.
I added the image processing tests, these are basically the same as SuperGlue's and LightGlue's with small differences to the keypoint matching mocks.
Also I refactored the modeling code to move the functions the closest to where they are called

run-slow: efficientloftr

This comment contains run-slow, running the specified jobs: 

models: ['models/efficientloftr']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36355). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Ok, I made final corrections on my side (removed custom rope and used the "default" one), looks good now! @sbucaille thanks for iterating 🤗
> 
> A few things left:
> 
> * Please merge PR's on the Hub (rope config update and model card tags)
> * Make a proper model card

I'm currently on it

> * Decide whether we are going to move the model to some org / community org

About that, I tried contacting the authors of MatchAnything (newer version of EfficientLoFTR and the same lab) and had no answer from them, I can send another email but maybe having one of you guys from HF push a little would work better. Otherwise there are some organization existing on the hub, the lab name is : ZJU3DV from Zhejiang University 

We can make a "-community" org as usual to have more control over checkpoints then

P.S. https://huggingface.co/organizations/zju-community (invited you)

Model card is ready, I switched the model repos, also switched to accelerate device in the convert script since I don't have a CUDA gpu anymore, tests should pass, anything else in mind ? 🤔 

> I switched the model repos, also switched to accelerate device in the convert script since I don't have a CUDA gpu anymore

I would recommend using a simpler way for conversion script

```py
device = "cuda" if torch.cuda.is_available else "CPU"
```

other than that, I think we are ready to merge, let me ping Arthur for approval

run-slow: efficientloftr

This comment contains run-slow, running the specified jobs: 

models: ['models/efficientloftr']
quantizations: [] ...

run-slow: efficientloftr

This comment contains run-slow, running the specified jobs: 

models: ['models/efficientloftr']
quantizations: [] ...

The refactoring changed weight parameters, let me fix that

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, efficientloftr, lightglue, superglue, superpoint

Problem fixed, I had to work around the `utils/check_config_attributes.py` check and rename `stage_out_channels` to `out_features` which is included in a list of attribute to be present and not used so the check does not fail.
Updated config on hub and tests accordingly

run-slow: efficientloftr

This comment contains run-slow, running the specified jobs: 

models: ['models/efficientloftr']
quantizations: [] ...

@sbucaille congrats on another model merged 🎉 🎉 🎉  Great work on building keypoint matching task in transformers 🚀