Add files using large-upload tool

README.md

@@ -0,0 +1,137 @@
+---
+license: other
+license_name: custom-apple-license
+license_link: https://github.com/apple/ml-tic-clip/blob/main/LICENSE
+tags:
+- vision
+- zero-shot-image-classification
+datasets:
+- apple/TiC-DataComp
+---
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+This repository contains TiC-CLIP models trained on TiC-DataComp-Yearly with data from 2014 to 2022 using our modified OpenCLIP code.
+For additional information refer to our [GitHub repo](https://github.com/apple/ml-tic-clip).
+
+## Model Details
+
+### Model Description
+
+Keeping large foundation models up to date on latest data is inherently expensive.
+To avoid the prohibitive costs of constantly retraining, it is imperative to continually train these models.
+This problem is exacerbated by the lack of any large scale continual learning benchmarks or baselines.
+We introduce the first set of web-scale Time-Continual (TiC) benchmarks for training vision-language models:
+TiC-DataComp, TiC-YFCC, and TiC-Redcaps. TiC-DataComp, our largest dataset,
+contains over 12.7B timestamped image-text pairs spanning 9 years (2014-2022).
+We first use our benchmarks to curate various dynamic evaluations to measure temporal robustness of existing models.
+We show OpenAI's CLIP (trained on data up to 2020) loses ≈8% zero-shot accuracy on our curated retrieval task from 2021-2022 compared with more recently trained models in OpenCLIP repository.
+We then study how to efficiently train models on time-continuous data.
+We demonstrate that a simple rehearsal-based approach that continues training from the last checkpoint and replays old data reduces compute by 2.5× when compared to the standard practice of retraining from scratch.
+Code is available at [this https URL](https://github.com/apple/ml-tic-clip).
+
+
+
+- **Developed by:** Apple
+- **License:** See [LICENSE](https://github.com/apple/ml-tic-clip/blob/main/LICENSE)
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [ml-tic-clip GitHub repo](https://github.com/apple/ml-tic-clip)
+- **Paper:** [TiC-CLIP: Continual Training of CLIP Models, Garg, S., Farajtabar, M., Pouransari, H., Vemulapalli, R., Mehta, S., Tuzel, O., Shankar, V. and Faghri, F., International Conference on Learning Representations (ICLR), 2024.](https://arxiv.org/abs/2310.16226)
+
+## Uses
+
+Researchers can use TiC-CLIP pretrained models for faster design of continual learning methods by start from a pretrained checkpoint and continually train on the next year or next month data.
+
+## How to Get Started with the Model
+
+The models are compatible with DataComp evaluation suite and our patched version of DataComp for evaluation on TiC-DataComp-Retrieval and TiC-DataCompNet.
+The models can also be used to resume a training or as initialization for new training using OpenCLIP code.
+Please follow instructions in our [GitHub repo](https://github.com/apple/ml-tic-clip) to create the evaluation sets or follow [DataComp](https://github.com/mlfoundations/datacomp) for the standard evaluations on 38 datasets.
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+Please refer to Sections 2-3 of our [TiC-CLIP](https://github.com/apple/ml-tic-clip) paper.
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]

params/2017.yaml

@@ -0,0 +1,99 @@
+accum_freq: 1
+aug_cfg: {}
+batch_size: 1408
+beta1: 0.9
+beta2: 0.98
+checkpoint_path: /tmp/restart_2017/datacomp_xlarge-bestpool_restart_2017/checkpoints
+coca_caption_loss_weight: 2.0
+coca_contrastive_loss_weight: 1.0
+copy_codebase: False
+csv_caption_key: title
+csv_img_key: filepath
+csv_separator: 	
+dataset_resampled: True
+dataset_type: webdataset
+ddp_static_graph: True
+debug: False
+decay_fraction: 0.2
+delete_previous_checkpoint: False
+device: cuda:0
+dist_backend: nccl
+dist_url: env://
+distill: False
+distill_model: None
+distill_pretrained: None
+distributed: True
+epochs: 32
+epochs_cooldown: None
+eps: 1e-06
+force_custom_text: False
+force_image_size: None
+force_patch_dropout: None
+force_qk_norm: False
+force_qk_norm_eps: 1e-05
+force_quick_gelu: False
+gather_with_grad: True
+grad_checkpointing: True
+grad_clip_norm: None
+horovod: False
+image_mean: None
+image_std: None
+imagenet_v2: None
+imagenet_val: ../imagenet_validation
+is_iteration_based: True
+local_loss: True
+local_rank: 0
+lock_image: False
+lock_image_freeze_bn_stats: False
+lock_image_unlocked_groups: 0
+lock_text: False
+lock_text_freeze_layer_norm: False
+lock_text_unlocked_layers: 0
+log_every_n_steps: 100
+log_level: 20
+log_local: False
+log_path: /tmp/restart_2017/datacomp_xlarge-bestpool_restart_2017/out.log
+logs: /tmp/restart_2017
+lr: 0.001
+lr_cooldown_end: 0.0
+lr_cooldown_power: 1.0
+lr_scheduler: cosine
+max_iterations: 20500
+model: ViT-B-16
+name: datacomp_xlarge-bestpool_restart_2017
+new_run: True
+no_set_device_rank: False
+precision: amp
+pretrained: 
+pretrained_image: False
+rank: 0
+remote_sync: xlarge_CL_bestpool_filter/restart_2017/
+remote_sync_frequency: 300
+remote_sync_protocol: s3
+report_to: 
+resume: xlarge_CL_bestpool_filter/cumulative_sequential_2016/datacomp_xlarge-bestpool-2023-09-12_08-24-13_cumulative_seq_2016/checkpoints/epoch_12.pt
+save_frequency: 1
+save_most_recent: True
+seed: 0
+skip_scheduler: False
+tensorboard: False
+tensorboard_path: 
+torchcompile: False
+torchscript: False
+trace: False
+train_data: xlarge_bestpool_filter/2017/0/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/1/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/10/{00000000..00000319}.tar::xlarge_bestpool_filter/2017/11/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/12/{00000000..00000359}.tar::xlarge_bestpool_filter/2017/13/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/14/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/15/{00000000..00000369}.tar::xlarge_bestpool_filter/2017/16/{00000000..00000331}.tar::xlarge_bestpool_filter/2017/17/{00000000..00000319}.tar::xlarge_bestpool_filter/2017/18/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/19/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/2/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/20/{00000000..00000372}.tar::xlarge_bestpool_filter/2017/21/{00000000..00000319}.tar::xlarge_bestpool_filter/2017/22/{00000000..00000319}.tar::xlarge_bestpool_filter/2017/23/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/24/{00000000..00000255}.tar::xlarge_bestpool_filter/2017/25/{00000000..00000255}.tar::xlarge_bestpool_filter/2017/26/{00000000..00000063}.tar::xlarge_bestpool_filter/2017/27/{00000000..00000107}.tar::xlarge_bestpool_filter/2017/28/{00000000..00000447}.tar::xlarge_bestpool_filter/2017/29/{00000000..00000447}.tar::xlarge_bestpool_filter/2017/3/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/30/{00000000..00000399}.tar::xlarge_bestpool_filter/2017/31/{00000000..00000399}.tar::xlarge_bestpool_filter/2017/32/{00000000..00000399}.tar::xlarge_bestpool_filter/2017/33/{00000000..00000401}.tar::xlarge_bestpool_filter/2017/34/{00000000..00000441}.tar::xlarge_bestpool_filter/2017/35/{00000000..00000433}.tar::xlarge_bestpool_filter/2017/36/{00000000..00000447}.tar::xlarge_bestpool_filter/2017/37/{00000000..00000447}.tar::xlarge_bestpool_filter/2017/38/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/39/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/4/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/40/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/41/{00000000..00000319}.tar::xlarge_bestpool_filter/2017/42/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/43/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/44/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/45/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/46/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/47/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/48/{00000000..00000419}.tar::xlarge_bestpool_filter/2017/49/{00000000..00000409}.tar::xlarge_bestpool_filter/2017/5/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/50/{00000000..00000447}.tar::xlarge_bestpool_filter/2017/51/{00000000..00000447}.tar::xlarge_bestpool_filter/2017/52/{00000000..00000447}.tar::xlarge_bestpool_filter/2017/53/{00000000..00000664}.tar::xlarge_bestpool_filter/2017/54/{00000000..00000639}.tar::xlarge_bestpool_filter/2017/55/{00000000..00000238}.tar::xlarge_bestpool_filter/2017/56/{00000000..00000237}.tar::xlarge_bestpool_filter/2017/57/{00000000..00000205}.tar::xlarge_bestpool_filter/2017/58/{00000000..00000191}.tar::xlarge_bestpool_filter/2017/59/{00000000..00000191}.tar::xlarge_bestpool_filter/2017/6/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/7/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/8/{00000000..00000383}.tar::xlarge_bestpool_filter/2017/9/{00000000..00000340}.tar
+train_data_upsampling_factors: None
+train_num_samples: None
+use_bn_sync: False
+use_bnb_linear: None
+val_data: None
+val_frequency: 1
+val_num_samples: None
+wandb: False
+wandb_notes: 
+wandb_project_name: open-clip
+warmup: 2000
+wd: 0.2
+workers: 4
+world_size: 64
+zeroshot_frequency: 1

params/2019.yaml

@@ -0,0 +1,99 @@
+accum_freq: 1
+aug_cfg: {}
+batch_size: 1408
+beta1: 0.9
+beta2: 0.98
+checkpoint_path: /tmp/restart_2019/datacomp_xlarge-bestpool_restart_2019/checkpoints
+coca_caption_loss_weight: 2.0
+coca_contrastive_loss_weight: 1.0
+copy_codebase: False
+csv_caption_key: title
+csv_img_key: filepath
+csv_separator: 	
+dataset_resampled: True
+dataset_type: webdataset
+ddp_static_graph: True
+debug: False
+decay_fraction: 0.2
+delete_previous_checkpoint: False
+device: cuda:0
+dist_backend: nccl
+dist_url: env://
+distill: False
+distill_model: None
+distill_pretrained: None
+distributed: True
+epochs: 32
+epochs_cooldown: None
+eps: 1e-06
+force_custom_text: False
+force_image_size: None
+force_patch_dropout: None
+force_qk_norm: False
+force_qk_norm_eps: 1e-05
+force_quick_gelu: False
+gather_with_grad: True
+grad_checkpointing: True
+grad_clip_norm: None
+horovod: False
+image_mean: None
+image_std: None
+imagenet_v2: None
+imagenet_val: ../imagenet_validation
+is_iteration_based: True
+local_loss: True
+local_rank: 0
+lock_image: False
+lock_image_freeze_bn_stats: False
+lock_image_unlocked_groups: 0
+lock_text: False
+lock_text_freeze_layer_norm: False
+lock_text_unlocked_layers: 0
+log_every_n_steps: 100
+log_level: 20
+log_local: False
+log_path: /tmp/restart_2019/datacomp_xlarge-bestpool_restart_2019/out.log
+logs: /tmp/restart_2019
+lr: 0.001
+lr_cooldown_end: 0.0
+lr_cooldown_power: 1.0
+lr_scheduler: cosine
+max_iterations: 20500
+model: ViT-B-16
+name: datacomp_xlarge-bestpool_restart_2019
+new_run: True
+no_set_device_rank: False
+precision: amp
+pretrained: 
+pretrained_image: False
+rank: 0
+remote_sync: xlarge_CL_bestpool_filter/restart_2019/
+remote_sync_frequency: 300
+remote_sync_protocol: s3
+report_to: 
+resume: xlarge_CL_bestpool_filter/restart_2018/datacomp_xlarge-bestpool_restart_2018/checkpoints/epoch_9.pt
+save_frequency: 1
+save_most_recent: True
+seed: 0
+skip_scheduler: False
+tensorboard: False
+tensorboard_path: 
+torchcompile: False
+torchscript: False
+trace: False
+train_data: xlarge_bestpool_filter/2019/0/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/1/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/10/{00000000..00000375}.tar::xlarge_bestpool_filter/2019/11/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/12/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/13/{00000000..00000385}.tar::xlarge_bestpool_filter/2019/14/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/15/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/16/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/17/{00000000..00000347}.tar::xlarge_bestpool_filter/2019/18/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/19/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/2/{00000000..00000399}.tar::xlarge_bestpool_filter/2019/20/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/21/{00000000..00000362}.tar::xlarge_bestpool_filter/2019/22/{00000000..00000319}.tar::xlarge_bestpool_filter/2019/23/{00000000..00000399}.tar::xlarge_bestpool_filter/2019/24/{00000000..00000279}.tar::xlarge_bestpool_filter/2019/25/{00000000..00000255}.tar::xlarge_bestpool_filter/2019/26/{00000000..00000063}.tar::xlarge_bestpool_filter/2019/27/{00000000..00000107}.tar::xlarge_bestpool_filter/2019/28/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/29/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/3/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/30/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/31/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/32/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/33/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/34/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/35/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/36/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/37/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/38/{00000000..00000412}.tar::xlarge_bestpool_filter/2019/39/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/4/{00000000..00000399}.tar::xlarge_bestpool_filter/2019/40/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/41/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/42/{00000000..00000395}.tar::xlarge_bestpool_filter/2019/43/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/44/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/45/{00000000..00000442}.tar::xlarge_bestpool_filter/2019/46/{00000000..00000431}.tar::xlarge_bestpool_filter/2019/47/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/48/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/49/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/5/{00000000..00000426}.tar::xlarge_bestpool_filter/2019/50/{00000000..00000439}.tar::xlarge_bestpool_filter/2019/51/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/52/{00000000..00000447}.tar::xlarge_bestpool_filter/2019/53/{00000000..00000703}.tar::xlarge_bestpool_filter/2019/54/{00000000..00000703}.tar::xlarge_bestpool_filter/2019/55/{00000000..00000255}.tar::xlarge_bestpool_filter/2019/56/{00000000..00000255}.tar::xlarge_bestpool_filter/2019/57/{00000000..00000255}.tar::xlarge_bestpool_filter/2019/58/{00000000..00000234}.tar::xlarge_bestpool_filter/2019/59/{00000000..00000247}.tar::xlarge_bestpool_filter/2019/6/{00000000..00000410}.tar::xlarge_bestpool_filter/2019/7/{00000000..00000436}.tar::xlarge_bestpool_filter/2019/8/{00000000..00000383}.tar::xlarge_bestpool_filter/2019/9/{00000000..00000383}.tar
+train_data_upsampling_factors: None
+train_num_samples: None
+use_bn_sync: False
+use_bnb_linear: None
+val_data: None
+val_frequency: 1
+val_num_samples: None
+wandb: False
+wandb_notes: 
+wandb_project_name: open-clip
+warmup: 2000
+wd: 0.2
+workers: 4
+world_size: 64
+zeroshot_frequency: 1

params/2020.yaml

@@ -0,0 +1,99 @@
+accum_freq: 1
+aug_cfg: {}
+batch_size: 1408
+beta1: 0.9
+beta2: 0.98
+checkpoint_path: /tmp/restart_2020/datacomp_xlarge-bestpool_restart_2020/checkpoints
+coca_caption_loss_weight: 2.0
+coca_contrastive_loss_weight: 1.0
+copy_codebase: False
+csv_caption_key: title
+csv_img_key: filepath
+csv_separator: 	
+dataset_resampled: True
+dataset_type: webdataset
+ddp_static_graph: True
+debug: False
+decay_fraction: 0.2
+delete_previous_checkpoint: False
+device: cuda:0
+dist_backend: nccl
+dist_url: env://
+distill: False
+distill_model: None
+distill_pretrained: None
+distributed: True
+epochs: 32
+epochs_cooldown: None
+eps: 1e-06
+force_custom_text: False
+force_image_size: None
+force_patch_dropout: None
+force_qk_norm: False
+force_qk_norm_eps: 1e-05
+force_quick_gelu: False
+gather_with_grad: True
+grad_checkpointing: True
+grad_clip_norm: None
+horovod: False
+image_mean: None
+image_std: None
+imagenet_v2: None
+imagenet_val: ../imagenet_validation
+is_iteration_based: True
+local_loss: True
+local_rank: 0
+lock_image: False
+lock_image_freeze_bn_stats: False
+lock_image_unlocked_groups: 0
+lock_text: False
+lock_text_freeze_layer_norm: False
+lock_text_unlocked_layers: 0
+log_every_n_steps: 100
+log_level: 20
+log_local: False
+log_path: /tmp/restart_2020/datacomp_xlarge-bestpool_restart_2020/out.log
+logs: /tmp/restart_2020
+lr: 0.001
+lr_cooldown_end: 0.0
+lr_cooldown_power: 1.0
+lr_scheduler: cosine
+max_iterations: 20500
+model: ViT-B-16
+name: datacomp_xlarge-bestpool_restart_2020
+new_run: True
+no_set_device_rank: False
+precision: amp
+pretrained: 
+pretrained_image: False
+rank: 0
+remote_sync: xlarge_CL_bestpool_filter/restart_2020/
+remote_sync_frequency: 300
+remote_sync_protocol: s3
+report_to: 
+resume: xlarge_CL_bestpool_filter/restart_2019/datacomp_xlarge-bestpool_restart_2019/checkpoints/epoch_9.pt
+save_frequency: 1
+save_most_recent: True
+seed: 0
+skip_scheduler: False
+tensorboard: False
+tensorboard_path: 
+torchcompile: False
+torchscript: False
+trace: False
+train_data: xlarge_bestpool_filter/2020/0/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/1/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/10/{00000000..00000278}.tar::xlarge_bestpool_filter/2020/11/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/12/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/13/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/14/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/15/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/16/{00000000..00000286}.tar::xlarge_bestpool_filter/2020/17/{00000000..00000266}.tar::xlarge_bestpool_filter/2020/18/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/19/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/2/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/20/{00000000..00000318}.tar::xlarge_bestpool_filter/2020/21/{00000000..00000273}.tar::xlarge_bestpool_filter/2020/22/{00000000..00000255}.tar::xlarge_bestpool_filter/2020/23/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/24/{00000000..00000253}.tar::xlarge_bestpool_filter/2020/25/{00000000..00000232}.tar::xlarge_bestpool_filter/2020/26/{00000000..00000063}.tar::xlarge_bestpool_filter/2020/27/{00000000..00000063}.tar::xlarge_bestpool_filter/2020/28/{00000000..00000383}.tar::xlarge_bestpool_filter/2020/29/{00000000..00000383}.tar::xlarge_bestpool_filter/2020/3/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/30/{00000000..00000329}.tar::xlarge_bestpool_filter/2020/31/{00000000..00000324}.tar::xlarge_bestpool_filter/2020/32/{00000000..00000323}.tar::xlarge_bestpool_filter/2020/33/{00000000..00000357}.tar::xlarge_bestpool_filter/2020/34/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/35/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/36/{00000000..00000377}.tar::xlarge_bestpool_filter/2020/37/{00000000..00000381}.tar::xlarge_bestpool_filter/2020/38/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/39/{00000000..00000318}.tar::xlarge_bestpool_filter/2020/4/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/40/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/41/{00000000..00000257}.tar::xlarge_bestpool_filter/2020/42/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/43/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/44/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/45/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/46/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/47/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/48/{00000000..00000383}.tar::xlarge_bestpool_filter/2020/49/{00000000..00000383}.tar::xlarge_bestpool_filter/2020/5/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/50/{00000000..00000383}.tar::xlarge_bestpool_filter/2020/51/{00000000..00000383}.tar::xlarge_bestpool_filter/2020/52/{00000000..00000377}.tar::xlarge_bestpool_filter/2020/53/{00000000..00000545}.tar::xlarge_bestpool_filter/2020/54/{00000000..00000511}.tar::xlarge_bestpool_filter/2020/55/{00000000..00000191}.tar::xlarge_bestpool_filter/2020/56/{00000000..00000191}.tar::xlarge_bestpool_filter/2020/57/{00000000..00000191}.tar::xlarge_bestpool_filter/2020/58/{00000000..00000191}.tar::xlarge_bestpool_filter/2020/59/{00000000..00000191}.tar::xlarge_bestpool_filter/2020/6/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/7/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/8/{00000000..00000319}.tar::xlarge_bestpool_filter/2020/9/{00000000..00000299}.tar
+train_data_upsampling_factors: None
+train_num_samples: None
+use_bn_sync: False
+use_bnb_linear: None
+val_data: None
+val_frequency: 1
+val_num_samples: None
+wandb: False
+wandb_notes: 
+wandb_project_name: open-clip
+warmup: 2000
+wd: 0.2
+workers: 4
+world_size: 64
+zeroshot_frequency: 1

params/2021.yaml

@@ -0,0 +1,99 @@
+accum_freq: 1
+aug_cfg: {}
+batch_size: 1408
+beta1: 0.9
+beta2: 0.98
+checkpoint_path: /tmp/restart_2021/datacomp_xlarge-bestpool_restart_2021/checkpoints
+coca_caption_loss_weight: 2.0
+coca_contrastive_loss_weight: 1.0
+copy_codebase: False
+csv_caption_key: title
+csv_img_key: filepath
+csv_separator: 	
+dataset_resampled: True
+dataset_type: webdataset
+ddp_static_graph: True
+debug: False
+decay_fraction: 0.2
+delete_previous_checkpoint: False
+device: cuda:0
+dist_backend: nccl
+dist_url: env://
+distill: False
+distill_model: None
+distill_pretrained: None
+distributed: True
+epochs: 32
+epochs_cooldown: None
+eps: 1e-06
+force_custom_text: False
+force_image_size: None
+force_patch_dropout: None
+force_qk_norm: False
+force_qk_norm_eps: 1e-05
+force_quick_gelu: False
+gather_with_grad: True
+grad_checkpointing: True
+grad_clip_norm: None
+horovod: False
+image_mean: None
+image_std: None
+imagenet_v2: None
+imagenet_val: ../imagenet_validation
+is_iteration_based: True
+local_loss: True
+local_rank: 0
+lock_image: False
+lock_image_freeze_bn_stats: False
+lock_image_unlocked_groups: 0
+lock_text: False
+lock_text_freeze_layer_norm: False
+lock_text_unlocked_layers: 0
+log_every_n_steps: 100
+log_level: 20
+log_local: False
+log_path: /tmp/restart_2021/datacomp_xlarge-bestpool_restart_2021/out.log
+logs: /tmp/restart_2021
+lr: 0.001
+lr_cooldown_end: 0.0
+lr_cooldown_power: 1.0
+lr_scheduler: cosine
+max_iterations: 20500
+model: ViT-B-16
+name: datacomp_xlarge-bestpool_restart_2021
+new_run: True
+no_set_device_rank: False
+precision: amp
+pretrained: 
+pretrained_image: False
+rank: 0
+remote_sync: xlarge_CL_bestpool_filter/restart_2021/
+remote_sync_frequency: 300
+remote_sync_protocol: s3
+report_to: 
+resume: xlarge_CL_bestpool_filter/restart_2020/datacomp_xlarge-bestpool_restart_2020/checkpoints/epoch_12.pt
+save_frequency: 1
+save_most_recent: True
+seed: 0
+skip_scheduler: False
+tensorboard: False
+tensorboard_path: 
+torchcompile: False
+torchscript: False
+trace: False
+train_data: xlarge_bestpool_filter/2021/0/{00000000..00000320}.tar::xlarge_bestpool_filter/2021/1/{00000000..00000321}.tar::xlarge_bestpool_filter/2021/10/{00000000..00000319}.tar::xlarge_bestpool_filter/2021/11/{00000000..00000321}.tar::xlarge_bestpool_filter/2021/12/{00000000..00000326}.tar::xlarge_bestpool_filter/2021/13/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/14/{00000000..00000363}.tar::xlarge_bestpool_filter/2021/15/{00000000..00000319}.tar::xlarge_bestpool_filter/2021/16/{00000000..00000319}.tar::xlarge_bestpool_filter/2021/17/{00000000..00000319}.tar::xlarge_bestpool_filter/2021/18/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/19/{00000000..00000334}.tar::xlarge_bestpool_filter/2021/2/{00000000..00000382}.tar::xlarge_bestpool_filter/2021/20/{00000000..00000319}.tar::xlarge_bestpool_filter/2021/21/{00000000..00000319}.tar::xlarge_bestpool_filter/2021/22/{00000000..00000319}.tar::xlarge_bestpool_filter/2021/23/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/24/{00000000..00000255}.tar::xlarge_bestpool_filter/2021/25/{00000000..00000255}.tar::xlarge_bestpool_filter/2021/26/{00000000..00000063}.tar::xlarge_bestpool_filter/2021/27/{00000000..00000063}.tar::xlarge_bestpool_filter/2021/28/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/29/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/3/{00000000..00000380}.tar::xlarge_bestpool_filter/2021/30/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/31/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/32/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/33/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/34/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/35/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/36/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/37/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/38/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/39/{00000000..00000373}.tar::xlarge_bestpool_filter/2021/4/{00000000..00000356}.tar::xlarge_bestpool_filter/2021/40/{00000000..00000357}.tar::xlarge_bestpool_filter/2021/41/{00000000..00000319}.tar::xlarge_bestpool_filter/2021/42/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/43/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/44/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/45/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/46/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/47/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/48/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/49/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/5/{00000000..00000382}.tar::xlarge_bestpool_filter/2021/50/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/51/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/52/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/53/{00000000..00000639}.tar::xlarge_bestpool_filter/2021/54/{00000000..00000599}.tar::xlarge_bestpool_filter/2021/55/{00000000..00000191}.tar::xlarge_bestpool_filter/2021/56/{00000000..00000191}.tar::xlarge_bestpool_filter/2021/57/{00000000..00000191}.tar::xlarge_bestpool_filter/2021/58/{00000000..00000191}.tar::xlarge_bestpool_filter/2021/59/{00000000..00000191}.tar::xlarge_bestpool_filter/2021/6/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/7/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/8/{00000000..00000383}.tar::xlarge_bestpool_filter/2021/9/{00000000..00000319}.tar
+train_data_upsampling_factors: None
+train_num_samples: None
+use_bn_sync: False
+use_bnb_linear: None
+val_data: None
+val_frequency: 1
+val_num_samples: None
+wandb: False
+wandb_notes: 
+wandb_project_name: open-clip
+warmup: 2000
+wd: 0.2
+workers: 4
+world_size: 64
+zeroshot_frequency: 1

params/2022.yaml

@@ -0,0 +1,99 @@
+accum_freq: 1
+aug_cfg: {}
+batch_size: 1408
+beta1: 0.9
+beta2: 0.98
+checkpoint_path: /tmp/restart_2022/datacomp_xlarge-bestpool_restart_2022/checkpoints
+coca_caption_loss_weight: 2.0
+coca_contrastive_loss_weight: 1.0
+copy_codebase: False
+csv_caption_key: title
+csv_img_key: filepath
+csv_separator: 	
+dataset_resampled: True
+dataset_type: webdataset
+ddp_static_graph: True
+debug: False
+decay_fraction: 0.2
+delete_previous_checkpoint: False
+device: cuda:0
+dist_backend: nccl
+dist_url: env://
+distill: False
+distill_model: None
+distill_pretrained: None
+distributed: True
+epochs: 32
+epochs_cooldown: None
+eps: 1e-06
+force_custom_text: False
+force_image_size: None
+force_patch_dropout: None
+force_qk_norm: False
+force_qk_norm_eps: 1e-05
+force_quick_gelu: False
+gather_with_grad: True
+grad_checkpointing: True
+grad_clip_norm: None
+horovod: False
+image_mean: None
+image_std: None
+imagenet_v2: None
+imagenet_val: ../imagenet_validation
+is_iteration_based: True
+local_loss: True
+local_rank: 0
+lock_image: False
+lock_image_freeze_bn_stats: False
+lock_image_unlocked_groups: 0
+lock_text: False
+lock_text_freeze_layer_norm: False
+lock_text_unlocked_layers: 0
+log_every_n_steps: 100
+log_level: 20
+log_local: False
+log_path: /tmp/restart_2022/datacomp_xlarge-bestpool_restart_2022/out.log
+logs: /tmp/restart_2022
+lr: 0.001
+lr_cooldown_end: 0.0
+lr_cooldown_power: 1.0
+lr_scheduler: cosine
+max_iterations: 20500
+model: ViT-B-16
+name: datacomp_xlarge-bestpool_restart_2022
+new_run: True
+no_set_device_rank: False
+precision: amp
+pretrained: 
+pretrained_image: False
+rank: 0
+remote_sync: xlarge_CL_bestpool_filter/restart_2022/
+remote_sync_frequency: 300
+remote_sync_protocol: s3
+report_to: 
+resume: xlarge_CL_bestpool_filter/restart_2021/datacomp_xlarge-bestpool_restart_2021/checkpoints/epoch_10.pt
+save_frequency: 1
+save_most_recent: True
+seed: 0
+skip_scheduler: False
+tensorboard: False
+tensorboard_path: 
+torchcompile: False
+torchscript: False
+trace: False
+train_data: xlarge_bestpool_filter/2022/0/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/1/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/10/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/11/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/12/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/13/{00000000..00000257}.tar::xlarge_bestpool_filter/2022/14/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/15/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/16/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/17/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/18/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/19/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/2/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/20/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/21/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/22/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/23/{00000000..00000289}.tar::xlarge_bestpool_filter/2022/24/{00000000..00000191}.tar::xlarge_bestpool_filter/2022/25/{00000000..00000191}.tar::xlarge_bestpool_filter/2022/26/{00000000..00000063}.tar::xlarge_bestpool_filter/2022/27/{00000000..00000063}.tar::xlarge_bestpool_filter/2022/28/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/29/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/3/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/30/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/31/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/32/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/33/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/34/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/35/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/36/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/37/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/38/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/39/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/4/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/40/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/41/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/42/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/43/{00000000..00000303}.tar::xlarge_bestpool_filter/2022/44/{00000000..00000306}.tar::xlarge_bestpool_filter/2022/45/{00000000..00000258}.tar::xlarge_bestpool_filter/2022/46/{00000000..00000259}.tar::xlarge_bestpool_filter/2022/47/{00000000..00000276}.tar::xlarge_bestpool_filter/2022/48/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/49/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/5/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/50/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/51/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/52/{00000000..00000319}.tar::xlarge_bestpool_filter/2022/53/{00000000..00000447}.tar::xlarge_bestpool_filter/2022/54/{00000000..00000447}.tar::xlarge_bestpool_filter/2022/55/{00000000..00000189}.tar::xlarge_bestpool_filter/2022/56/{00000000..00000188}.tar::xlarge_bestpool_filter/2022/57/{00000000..00000184}.tar::xlarge_bestpool_filter/2022/58/{00000000..00000163}.tar::xlarge_bestpool_filter/2022/59/{00000000..00000176}.tar::xlarge_bestpool_filter/2022/6/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/7/{00000000..00000310}.tar::xlarge_bestpool_filter/2022/8/{00000000..00000255}.tar::xlarge_bestpool_filter/2022/9/{00000000..00000255}.tar
+train_data_upsampling_factors: None
+train_num_samples: None
+use_bn_sync: False
+use_bnb_linear: None
+val_data: None
+val_frequency: 1
+val_num_samples: None
+wandb: False
+wandb_notes: 
+wandb_project_name: open-clip
+warmup: 2000
+wd: 0.2
+workers: 4
+world_size: 64
+zeroshot_frequency: 1