Zero-Shot Image Classification
TiC-CLIP
vision
fartashf commited on
Commit
c6ab028
1 Parent(s): b20ffd5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +28 -2
README.md CHANGED
@@ -55,12 +55,14 @@ The models can also be used to resume a training or as initialization for new tr
55
  Please follow instructions in our [GitHub repo](https://github.com/apple/ml-tic-clip) to create the evaluation sets or follow [DataComp](https://github.com/mlfoundations/datacomp) for the standard evaluations on 38 datasets.
56
 
57
  The following snippet assumes the TiC-DataComp data has been prepared and following the instructions in the GitHub repo.
 
 
58
  ```bash
59
  YEAR=2016 # There are no models before 2016 since data from 2014-2016 were compined into one year
60
  REPO="apple/TiC-CLIP-basic-cumulative"
61
  huggingface-cli download $REPO checkpoints/$YEAR.pt
62
 
63
- ## Train Cummulative
64
  pushd datacomp
65
  final_data_dir=$TIC_DATACOMP_Y_PATH/train/$YEAR/
66
  torchrun --nproc_per_node 8 --nnodes 1 \
@@ -74,8 +76,10 @@ torchrun --nproc_per_node 8 --nnodes 1 \
74
  --save_frequency 1 \
75
  --resume
76
  popd
 
77
 
78
- ## Evaluate Model
 
79
  # Evaluate a ViT-B/16 model on TiC/Retrieval/Yearly/$YEAR and
80
  # TiC/DataCompNet/Yearly/$YEAR
81
  pushd datacomp
@@ -83,6 +87,28 @@ python ../dataset_creation/tic-datacomp/generate_tasklist.py --yaml-path tasklis
83
  python evaluate.py --data_dir data/ --train_output_dir ./results --use_model "ViT-B-16 $YEAR.pt" --skip_hf --skip_db --skip_notification
84
  ```
85
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
  ## Training Details
87
 
88
  ### Training Data
 
55
  Please follow instructions in our [GitHub repo](https://github.com/apple/ml-tic-clip) to create the evaluation sets or follow [DataComp](https://github.com/mlfoundations/datacomp) for the standard evaluations on 38 datasets.
56
 
57
  The following snippet assumes the TiC-DataComp data has been prepared and following the instructions in the GitHub repo.
58
+
59
+ ### Training
60
  ```bash
61
  YEAR=2016 # There are no models before 2016 since data from 2014-2016 were compined into one year
62
  REPO="apple/TiC-CLIP-basic-cumulative"
63
  huggingface-cli download $REPO checkpoints/$YEAR.pt
64
 
65
+ ## Train
66
  pushd datacomp
67
  final_data_dir=$TIC_DATACOMP_Y_PATH/train/$YEAR/
68
  torchrun --nproc_per_node 8 --nnodes 1 \
 
76
  --save_frequency 1 \
77
  --resume
78
  popd
79
+ ```
80
 
81
+ ### Evaluation
82
+ ```bash
83
  # Evaluate a ViT-B/16 model on TiC/Retrieval/Yearly/$YEAR and
84
  # TiC/DataCompNet/Yearly/$YEAR
85
  pushd datacomp
 
87
  python evaluate.py --data_dir data/ --train_output_dir ./results --use_model "ViT-B-16 $YEAR.pt" --skip_hf --skip_db --skip_notification
88
  ```
89
 
90
+ ### OpenCLIP Load and Inference Example
91
+ ```python
92
+ import open_clip
93
+ from huggingface_hub import hf_hub_download
94
+ filename = hf_hub_download(repo_id="apple/TiC-CLIP-basic-cumulative", filename="checkpoints/2016.pt")
95
+ model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-16', filename)
96
+ tokenizer = open_clip.get_tokenizer('ViT-B-16')
97
+
98
+ image = preprocess(Image.open("image.png").convert('RGB')).unsqueeze(0)
99
+ text = tokenizer(["a diagram", "a dog", "a cat"])
100
+
101
+ with torch.no_grad(), torch.cuda.amp.autocast():
102
+ image_features = model.encode_image(image)
103
+ text_features = model.encode_text(text)
104
+ image_features /= image_features.norm(dim=-1, keepdim=True)
105
+ text_features /= text_features.norm(dim=-1, keepdim=True)
106
+
107
+ text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
108
+
109
+ print("Label probs:", text_probs)
110
+ ```
111
+
112
  ## Training Details
113
 
114
  ### Training Data