VicenteVivanco/geoclip-geolocation

GeoCLIP: Worldwide Image Geolocation

GeoCLIP estimates where a photo was taken. It embeds images and GPS coordinates in a shared space, then ranks a gallery of candidate coordinates by their similarity to the photo. It returns the highest-ranked locations and their scores.

We presented GeoCLIP at NeurIPS 2023. This page holds the trained weights and the 100,000-coordinate gallery used for prediction. The Python package below includes the same components, so you can start with either.

Demo · Paper · GitHub · Python package

Download weights · Browse all model files

Get started

pip install geoclip

from geoclip import GeoCLIP

model = GeoCLIP().eval()
coordinates, scores = model.predict("image.jpg", top_k=5)
for (latitude, longitude), score in zip(coordinates, scores):
    print(f"({latitude:.4f}, {longitude:.4f}) — score: {score:.4f}")

The package includes the trained GeoCLIP components and a gallery of 100,000 GPS coordinates. The frozen OpenAI CLIP ViT-L/14 backbone is a separate download, fetched on first use if it is not already cached. For GPS features to use in your own models, see the GeoCLIP Location Encoder.

Load from Hugging Face

The from_pretrained helpers are available in geoclip 1.2.2 and newer.

python -m pip install --upgrade "geoclip>=1.2.2"

from geoclip import GeoCLIP

model = GeoCLIP.from_pretrained("VicenteVivanco/geoclip-geolocation")
coordinates, scores = model.predict("image.jpg", top_k=5)
print(coordinates, scores)

This loads the trained GeoCLIP weights and coordinate gallery from this repository, plus the frozen CLIP backbone at the revision recorded in config.json. Downloads are cached. You can pass revision to select a Hub commit or a local release folder in place of the model ID.

Manual Hub loading with geoclip 1.2.1

The quick start uses the weights included in the Python package. To load the weights and coordinate gallery from this page instead, use the example below.

pip install geoclip huggingface_hub safetensors

import json
import pandas as pd
import torch
from geoclip import GeoCLIP
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoProcessor, CLIPModel

folder = snapshot_download("VicenteVivanco/geoclip-geolocation")
config = json.load(open(f"{folder}/config.json"))
weights = load_file(f"{folder}/model.safetensors")
model = GeoCLIP(from_pretrained=False)
model.image_encoder.CLIP = CLIPModel.from_pretrained(
    config["backbone"]["repo_id"], revision=config["backbone"]["revision"]
)
model.image_encoder.image_processor = AutoProcessor.from_pretrained(
    config["backbone"]["repo_id"], revision=config["backbone"]["revision"]
)
model.image_encoder.mlp.load_state_dict({
    key.removeprefix("image_encoder.mlp."): value
    for key, value in weights.items() if key.startswith("image_encoder.mlp.")
})
model.location_encoder.load_state_dict({
    key.removeprefix("location_encoder."): value
    for key, value in weights.items() if key.startswith("location_encoder.")
})
with torch.no_grad():
    model.logit_scale.copy_(weights["logit_scale"])
gallery = pd.read_csv(f"{folder}/coordinates_100K.csv")[["LAT", "LON"]]
model.gps_gallery = torch.tensor(gallery.values, dtype=torch.float32)
model.eval()
coordinates, scores = model.predict("image.jpg", top_k=5)
print(coordinates, scores)

How it works

We trained GeoCLIP contrastively on roughly 4.7 million image–GPS pairs from the MP-16 dataset. As in OpenAI's CLIP, matching pairs are brought together in a shared embedding space. The image encoder uses a frozen CLIP ViT-L/14 backbone followed by a learned projection head. The location encoder maps latitude and longitude to vectors in the same space.

To predict a location, GeoCLIP compares the photo's embedding with the gallery's location embeddings and returns the best matches.

Understanding the predictions

Predictions come from the gallery, so the model can only return coordinates it was given. Accuracy depends on the photo and how well the gallery covers the region where it was taken. Scores rank the candidates relative to one another; they are not probabilities that a prediction is correct.

The package and demo use a single processed image and a 100K-coordinate gallery. The paper evaluates additional crop and gallery settings, so these examples do not reproduce its full evaluation protocol.

What's included in the download

• model.safetensors: the trained image projection head, location encoder (including its frozen Fourier matrices), and logit scale.
• coordinates_100K.csv: the gallery of candidate GPS coordinates.
• config.json: model settings and the pinned revision of the separate openai/clip-vit-large-patch14 backbone.
• provenance.json: the source commit, package version, and SHA-256 hashes.

The exported tensors match the original checkpoints exactly. The source checkpoints and coordinate gallery match those included in geoclip==1.2.1.

Citation and license

If GeoCLIP is useful in your work, please cite the paper:

@inproceedings{geoclip,
  title={GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization},
  author={Vivanco Cepeda, Vicente and Nayak, Gaurav Kumar and Shah, Mubarak},
  booktitle={Advances in Neural Information Processing Systems},
  year={2023}
}

Released under the MIT license. GeoCLIP builds on OpenAI's CLIP and includes code from Joshua M. Long's Random Fourier Features PyTorch.