Add AstroCLIP model support
Author: giovannicozzolongo
Pull request: no

### Feature request

I'd like to add AstroCLIP to Transformers.

AstroCLIP is a CLIP-style dual encoder for astronomy data. It aligns galaxy image
cutouts with galaxy spectra using a DINOv2 image encoder, a SpecFormer spectrum
encoder, cross-attention pooling heads and a symmetric contrastive loss.

Paper: https://huggingface.co/papers/2310.03024

The implementation I am preparing includes:

- `AstroClipConfig` and `AstroClipSpectrumConfig`
- `AstroClipModel` and `SpecFormerModel`
- image and spectrum preprocessing classes
- `AstroClipProcessor`
- Auto class registration
- docs and model tests
- a checkpoint conversion script for the upstream Lightning checkpoints

Validation against the upstream checkpoints:

- SpecFormer: 79 mapped keys, 0 skipped, strict load passes.
- AstroCLIP: 446 mapped source keys, 542 target tensors, 0 skipped, strict load
  passes (542/542 model keys).
- The converted model reloads via `AutoProcessor.from_pretrained` and
  `AutoModel.from_pretrained`.
- Fixed-input parity against the upstream implementation passes at `rtol=1e-4`,
  `atol=1e-5` for image embeddings, spectrum embeddings, logits and contrastive
  loss.

Before opening a PR: does AstroCLIP fit the current model scope and would you
prefer the work split (e.g. model implementation first, checkpoint/model-card
publication separately)?

Comments:
Hmmn, the first question I'd ask is whether we need a model PR for this, or whether the architecture is similar enough to an existing model that we can reuse that. Secondly, I'm not an astrophysicist, but the model seems to be a few years old. Has it gotten significant usage in the community in that time?

I do not think AstroCLIP maps cleanly onto an existing model class. The image branch can reuse `Dinov2Model`, and that is the approach I would take. The second branch works on galaxy spectra: it uses a SpecFormer encoder with its own preprocessing, learned embeddings, reconstruction head and checkpoint layout. So `CLIPModel` or `VisionTextDualEncoder` would cover the CLIP-style loss, but not the spectrum encoder or the checkpoint-compatible structure. I see AstroCLIP as a specialized scientific model with a modest but real user base. The evidence I found is:

- arXiv 2023, MNRAS 2024
- Crossref currently shows 61 citations for the MNRAS paper
- the official PolymathicAI/AstroCLIP repo has 179 stars and 27 forks
- there is already a `polymathic-ai/astroclip` checkpoint on the Hub, but not in native Transformers format

Hmn, this might be a little too low to integrate into the main library! What I'd suggest instead is making a remote code model, take a look [here](https://huggingface.co/docs/transformers/custom_models). Remote code models include their code in the Hub repo, and can be loaded with all the normal classes like `AutoModel`, users just have to specify `trust_remote_code=True`. Plenty of big models like Phi launch as remote code.

If a remote code model gets a lot of usage, we can migrate it into the main library later. Once it's in the main library the transformers team takes responsibility for maintaining it indefinitely, so we don't want to merge models that won't get used!

Published [here](https://huggingface.co/giovannicozzolongo/astroclip)

Nice! I'll close the issue for now, but the model looks cool, and I hope it's useful to the astrophysics community. If it becomes big enough that we should take over maintenance feel free to ping me again to reopen!