regisss HF staff commited on
Commit
7bff1e9
1 Parent(s): be6274d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +87 -0
README.md CHANGED
@@ -1,3 +1,90 @@
1
  ---
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
4
+
5
+ [Optimum Habana](https://github.com/huggingface/optimum-habana) is the interface between the Hugging Face Transformers and Diffusers libraries and Habana's Gaudi processor (HPU).
6
+ It provides a set of tools enabling easy and fast model loading, training and inference on single- and multi-HPU settings for different downstream tasks.
7
+ Learn more about how to take advantage of the power of Habana HPUs to train and deploy Transformers and Diffusers models at [hf.co/hardware/habana](https://huggingface.co/hardware/habana).
8
+
9
+ ## CLIP model HPU configuration
10
+
11
+ This model only contains the `GaudiConfig` file for running CLIP-like models (e.g. [this one](https://huggingface.co/openai/clip-vit-large-patch14)) on Habana's Gaudi processors (HPU).
12
+
13
+ **This model contains no model weights, only a GaudiConfig.**
14
+
15
+ This enables to specify:
16
+ - `use_habana_mixed_precision`: whether to use Habana Mixed Precision (HMP)
17
+ - `hmp_opt_level`: optimization level for HMP, see [here](https://docs.habana.ai/en/latest/PyTorch/PyTorch_Mixed_Precision/PT_Mixed_Precision.html#configuration-options) for a detailed explanation
18
+ - `hmp_bf16_ops`: list of operators that should run in bf16
19
+ - `hmp_fp32_ops`: list of operators that should run in fp32
20
+ - `hmp_is_verbose`: verbosity
21
+ - `use_fused_adam`: whether to use Habana's custom AdamW implementation
22
+ - `use_fused_clip_norm`: whether to use Habana's fused gradient norm clipping operator
23
+
24
+ ## Usage
25
+
26
+ The model is instantiated the same way as in the Transformers library.
27
+ The only difference is that there are a few new training arguments specific to HPUs.
28
+
29
+ [Here](https://github.com/huggingface/optimum-habana/blob/main/examples/contrastive-image-text) is an example script to fine-tune a model on COCO.
30
+ Use it as follows:
31
+
32
+ 1. You first need to download the dataset:
33
+ ```bash
34
+ mkdir data
35
+ cd data
36
+ wget http://images.cocodataset.org/zips/train2017.zip
37
+ wget http://images.cocodataset.org/zips/val2017.zip
38
+ wget http://images.cocodataset.org/zips/test2017.zip
39
+ wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
40
+ wget http://images.cocodataset.org/annotations/image_info_test2017.zip
41
+ cd ..
42
+ ```
43
+
44
+ 2. Then, you can create a model from pretrained vision and text decoder models:
45
+ ```python
46
+ from transformers import (
47
+ VisionTextDualEncoderModel,
48
+ VisionTextDualEncoderProcessor,
49
+ AutoTokenizer,
50
+ AutoImageProcessor
51
+ )
52
+
53
+ model = VisionTextDualEncoderModel.from_vision_text_pretrained(
54
+ "openai/clip-vit-large-patch14", "roberta-large"
55
+ )
56
+
57
+ tokenizer = AutoTokenizer.from_pretrained("roberta-large")
58
+ image_processor = AutoImageProcessor.from_pretrained("openai/clip-vit-large-patch14")
59
+ processor = VisionTextDualEncoderProcessor(image_processor, tokenizer)
60
+
61
+ # save the model and processor
62
+ model.save_pretrained("clip-roberta")
63
+ processor.save_pretrained("clip-roberta")
64
+ ```
65
+
66
+ 3. Finally, you can run it with the following command:
67
+ ```bash
68
+ python run_clip.py \
69
+ --output_dir ./clip-roberta-finetuned \
70
+ --model_name_or_path ./clip-roberta \
71
+ --data_dir $PWD/data \
72
+ --dataset_name ydshieh/coco_dataset_script \
73
+ --dataset_config_name=2017 \
74
+ --image_column image_path \
75
+ --caption_column caption \
76
+ --remove_unused_columns=False \
77
+ --do_train --do_eval \
78
+ --per_device_train_batch_size="16" \
79
+ --per_device_eval_batch_size="16" \
80
+ --learning_rate="5e-5" --warmup_steps="0" --weight_decay 0.1 \
81
+ --overwrite_output_dir \
82
+ --save_strategy epoch \
83
+ --use_habana \
84
+ --use_lazy_mode \
85
+ --use_hpu_graphs \
86
+ --gaudi_config_name Habana/clip \
87
+ --throughput_warmup_steps 2
88
+ ```
89
+
90
+ Check the [documentation](https://huggingface.co/docs/optimum/habana/index) out for more advanced usage and examples.