Hugging Face's logo

aissatoubalde
/
lab

Text Generation
Transformers
Safetensors
phi
custom_code
Inference Endpoints
text-generation-inference
Model card Files Community
aissatoubalde commited on
Commit
182f594
•
1 Parent(s): a2a5a48

Training in progress, epoch 0

Browse files
Files changed (7) hide show
  1. .ipynb_checkpoints/Untitled-checkpoint.ipynb +6 -0
  2. .ipynb_checkpoints/phi-2-custom-2-checkpoint.ipynb +0 -0
  3. Untitled.ipynb +33 -0
  4. adapter_config.json +2 -0
  5. adapter_model.safetensors +3 -0
  6. phi-2-custom-2.ipynb +1202 -0
  7. training_args.bin +1 -1
.ipynb_checkpoints/Untitled-checkpoint.ipynb ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [],
3
+ "metadata": {},
4
+ "nbformat": 4,
5
+ "nbformat_minor": 5
6
+ }
.ipynb_checkpoints/phi-2-custom-2-checkpoint.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
Untitled.ipynb ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "code",
5
+ "execution_count": null,
6
+ "id": "e578b1ad-8a97-4984-8f23-ed977c286b99",
7
+ "metadata": {},
8
+ "outputs": [],
9
+ "source": []
10
+ }
11
+ ],
12
+ "metadata": {
13
+ "kernelspec": {
14
+ "display_name": "Python 3 (ipykernel)",
15
+ "language": "python",
16
+ "name": "python3"
17
+ },
18
+ "language_info": {
19
+ "codemirror_mode": {
20
+ "name": "ipython",
21
+ "version": 3
22
+ },
23
+ "file_extension": ".py",
24
+ "mimetype": "text/x-python",
25
+ "name": "python",
26
+ "nbconvert_exporter": "python",
27
+ "pygments_lexer": "ipython3",
28
+ "version": "3.11.5"
29
+ }
30
+ },
31
+ "nbformat": 4,
32
+ "nbformat_minor": 5
33
+ }
adapter_config.json CHANGED
@@ -19,6 +19,8 @@
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
 
 
22
  "v_proj",
23
  "q_proj"
24
  ],
 
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
22
+ "fc2",
23
+ "fc1",
24
  "v_proj",
25
  "q_proj"
26
  ],
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a15166ef18e36dd8d4dfe3000eed808ed96fabf11571a1ca15852d24f787cab5
3
+ size 73433680
phi-2-custom-2.ipynb ADDED
@@ -0,0 +1,1202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "code",
5
+ "execution_count": 1,
6
+ "id": "b3b92bc7-d105-405f-970d-804d298b9976",
7
+ "metadata": {
8
+ "scrolled": true
9
+ },
10
+ "outputs": [
11
+ {
12
+ "name": "stdout",
13
+ "output_type": "stream",
14
+ "text": [
15
+ "Collecting accelerate\n",
16
+ " Downloading accelerate-0.27.2-py3-none-any.whl.metadata (18 kB)\n",
17
+ "Collecting transformers\n",
18
+ " Downloading transformers-4.37.2-py3-none-any.whl.metadata (129 kB)\n",
19
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.4/129.4 kB\u001b[0m \u001b[31m41.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
20
+ "\u001b[?25hCollecting einops\n",
21
+ " Downloading einops-0.7.0-py3-none-any.whl.metadata (13 kB)\n",
22
+ "Collecting datasets\n",
23
+ " Downloading datasets-2.17.0-py3-none-any.whl.metadata (20 kB)\n",
24
+ "Collecting peft\n",
25
+ " Downloading peft-0.8.2-py3-none-any.whl.metadata (25 kB)\n",
26
+ "Collecting bitsandbytes\n",
27
+ " Downloading bitsandbytes-0.42.0-py3-none-any.whl.metadata (9.9 kB)\n",
28
+ "Collecting torch\n",
29
+ " Downloading torch-2.2.0-cp311-cp311-manylinux1_x86_64.whl.metadata (25 kB)\n",
30
+ "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.11/site-packages (from accelerate) (1.26.4)\n",
31
+ "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.11/site-packages (from accelerate) (23.2)\n",
32
+ "Requirement already satisfied: psutil in /usr/local/lib/python3.11/site-packages (from accelerate) (5.9.8)\n",
33
+ "Requirement already satisfied: pyyaml in /usr/local/lib/python3.11/site-packages (from accelerate) (6.0.1)\n",
34
+ "Collecting huggingface-hub (from accelerate)\n",
35
+ " Downloading huggingface_hub-0.20.3-py3-none-any.whl.metadata (12 kB)\n",
36
+ "Collecting safetensors>=0.3.1 (from accelerate)\n",
37
+ " Downloading safetensors-0.4.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)\n",
38
+ "Collecting filelock (from transformers)\n",
39
+ " Downloading filelock-3.13.1-py3-none-any.whl.metadata (2.8 kB)\n",
40
+ "Collecting regex!=2019.12.17 (from transformers)\n",
41
+ " Downloading regex-2023.12.25-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)\n",
42
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 kB\u001b[0m \u001b[31m422.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
43
+ "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.11/site-packages (from transformers) (2.31.0)\n",
44
+ "Collecting tokenizers<0.19,>=0.14 (from transformers)\n",
45
+ " Downloading tokenizers-0.15.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
46
+ "Collecting tqdm>=4.27 (from transformers)\n",
47
+ " Downloading tqdm-4.66.2-py3-none-any.whl.metadata (57 kB)\n",
48
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.6/57.6 kB\u001b[0m \u001b[31m207.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
49
+ "\u001b[?25hCollecting pyarrow>=12.0.0 (from datasets)\n",
50
+ " Downloading pyarrow-15.0.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.0 kB)\n",
51
+ "Collecting pyarrow-hotfix (from datasets)\n",
52
+ " Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB)\n",
53
+ "Collecting dill<0.3.9,>=0.3.0 (from datasets)\n",
54
+ " Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)\n",
55
+ "Collecting pandas (from datasets)\n",
56
+ " Downloading pandas-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)\n",
57
+ "Collecting xxhash (from datasets)\n",
58
+ " Downloading xxhash-3.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
59
+ "Collecting multiprocess (from datasets)\n",
60
+ " Downloading multiprocess-0.70.16-py311-none-any.whl.metadata (7.2 kB)\n",
61
+ "Collecting fsspec<=2023.10.0,>=2023.1.0 (from fsspec[http]<=2023.10.0,>=2023.1.0->datasets)\n",
62
+ " Downloading fsspec-2023.10.0-py3-none-any.whl.metadata (6.8 kB)\n",
63
+ "Requirement already satisfied: aiohttp in /usr/local/lib/python3.11/site-packages (from datasets) (3.8.3)\n",
64
+ "Collecting scipy (from bitsandbytes)\n",
65
+ " Downloading scipy-1.12.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)\n",
66
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.4/60.4 kB\u001b[0m \u001b[31m480.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
67
+ "\u001b[?25hRequirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.11/site-packages (from torch) (4.9.0)\n",
68
+ "Collecting sympy (from torch)\n",
69
+ " Downloading sympy-1.12-py3-none-any.whl.metadata (12 kB)\n",
70
+ "Collecting networkx (from torch)\n",
71
+ " Downloading networkx-3.2.1-py3-none-any.whl.metadata (5.2 kB)\n",
72
+ "Requirement already satisfied: jinja2 in /usr/local/lib/python3.11/site-packages (from torch) (3.1.3)\n",
73
+ "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch)\n",
74
+ " Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
75
+ "Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch)\n",
76
+ " Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
77
+ "Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch)\n",
78
+ " Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
79
+ "Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch)\n",
80
+ " Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
81
+ "Collecting nvidia-cublas-cu12==12.1.3.1 (from torch)\n",
82
+ " Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
83
+ "Collecting nvidia-cufft-cu12==11.0.2.54 (from torch)\n",
84
+ " Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
85
+ "Collecting nvidia-curand-cu12==10.3.2.106 (from torch)\n",
86
+ " Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
87
+ "Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch)\n",
88
+ " Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
89
+ "Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch)\n",
90
+ " Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
91
+ "Collecting nvidia-nccl-cu12==2.19.3 (from torch)\n",
92
+ " Downloading nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB)\n",
93
+ "Collecting nvidia-nvtx-cu12==12.1.105 (from torch)\n",
94
+ " Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)\n",
95
+ "Collecting triton==2.2.0 (from torch)\n",
96
+ " Downloading triton-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.4 kB)\n",
97
+ "Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch)\n",
98
+ " Downloading nvidia_nvjitlink_cu12-12.3.101-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
99
+ "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets) (23.2.0)\n",
100
+ "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets) (2.1.1)\n",
101
+ "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets) (6.0.5)\n",
102
+ "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets) (4.0.3)\n",
103
+ "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets) (1.9.4)\n",
104
+ "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets) (1.4.1)\n",
105
+ "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets) (1.3.1)\n",
106
+ "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/site-packages (from requests->transformers) (3.6)\n",
107
+ "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/site-packages (from requests->transformers) (2.2.0)\n",
108
+ "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/site-packages (from requests->transformers) (2024.2.2)\n",
109
+ "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/site-packages (from jinja2->torch) (2.1.5)\n",
110
+ "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/site-packages (from pandas->datasets) (2.8.2)\n",
111
+ "Collecting pytz>=2020.1 (from pandas->datasets)\n",
112
+ " Downloading pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)\n",
113
+ "Collecting tzdata>=2022.7 (from pandas->datasets)\n",
114
+ " Downloading tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB)\n",
115
+ "Collecting mpmath>=0.19 (from sympy->torch)\n",
116
+ " Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)\n",
117
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n",
118
+ "Downloading accelerate-0.27.2-py3-none-any.whl (279 kB)\n",
119
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m280.0/280.0 kB\u001b[0m \u001b[31m11.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
120
+ "\u001b[?25hDownloading transformers-4.37.2-py3-none-any.whl (8.4 MB)\n",
121
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.4/8.4 MB\u001b[0m \u001b[31m285.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
122
+ "\u001b[?25hDownloading einops-0.7.0-py3-none-any.whl (44 kB)\n",
123
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.6/44.6 kB\u001b[0m \u001b[31m444.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
124
+ "\u001b[?25hDownloading datasets-2.17.0-py3-none-any.whl (536 kB)\n",
125
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.6/536.6 kB\u001b[0m \u001b[31m484.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
126
+ "\u001b[?25hDownloading peft-0.8.2-py3-none-any.whl (183 kB)\n",
127
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m183.4/183.4 kB\u001b[0m \u001b[31m480.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
128
+ "\u001b[?25hDownloading bitsandbytes-0.42.0-py3-none-any.whl (105.0 MB)\n",
129
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m105.0/105.0 MB\u001b[0m \u001b[31m180.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
130
+ "\u001b[?25hDownloading torch-2.2.0-cp311-cp311-manylinux1_x86_64.whl (755.5 MB)\n",
131
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m755.5/755.5 MB\u001b[0m \u001b[31m220.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
132
+ "\u001b[?25hDownloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
133
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m410.6/410.6 MB\u001b[0m \u001b[31m168.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
134
+ "\u001b[?25hDownloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
135
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.1/14.1 MB\u001b[0m \u001b[31m130.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
136
+ "\u001b[?25hDownloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
137
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.7/23.7 MB\u001b[0m \u001b[31m238.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
138
+ "\u001b[?25hDownloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
139
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m823.6/823.6 kB\u001b[0m \u001b[31m490.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
140
+ "\u001b[?25hDownloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
141
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m731.7/731.7 MB\u001b[0m \u001b[31m216.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
142
+ "\u001b[?25hDownloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
143
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.6/121.6 MB\u001b[0m \u001b[31m262.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
144
+ "\u001b[?25hDownloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
145
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.5/56.5 MB\u001b[0m \u001b[31m143.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
146
+ "\u001b[?25hDownloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
147
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.2/124.2 MB\u001b[0m \u001b[31m137.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
148
+ "\u001b[?25hDownloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
149
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.0/196.0 MB\u001b[0m \u001b[31m288.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
150
+ "\u001b[?25hDownloading nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl (166.0 MB)\n",
151
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m166.0/166.0 MB\u001b[0m \u001b[31m266.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
152
+ "\u001b[?25hDownloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
153
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.1/99.1 kB\u001b[0m \u001b[31m446.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
154
+ "\u001b[?25hDownloading triton-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (167.9 MB)\n",
155
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m167.9/167.9 MB\u001b[0m \u001b[31m235.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
156
+ "\u001b[?25hDownloading dill-0.3.8-py3-none-any.whl (116 kB)\n",
157
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m352.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
158
+ "\u001b[?25hDownloading fsspec-2023.10.0-py3-none-any.whl (166 kB)\n",
159
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m166.4/166.4 kB\u001b[0m \u001b[31m496.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
160
+ "\u001b[?25hDownloading huggingface_hub-0.20.3-py3-none-any.whl (330 kB)\n",
161
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m330.1/330.1 kB\u001b[0m \u001b[31m548.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
162
+ "\u001b[?25hDownloading pyarrow-15.0.0-cp311-cp311-manylinux_2_28_x86_64.whl (38.3 MB)\n",
163
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m38.3/38.3 MB\u001b[0m \u001b[31m164.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
164
+ "\u001b[?25hDownloading regex-2023.12.25-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (785 kB)\n",
165
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m785.1/785.1 kB\u001b[0m \u001b[31m540.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
166
+ "\u001b[?25hDownloading safetensors-0.4.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
167
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m564.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
168
+ "\u001b[?25hDownloading tokenizers-0.15.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)\n",
169
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m167.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
170
+ "\u001b[?25hDownloading tqdm-4.66.2-py3-none-any.whl (78 kB)\n",
171
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.3/78.3 kB\u001b[0m \u001b[31m271.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
172
+ "\u001b[?25hDownloading filelock-3.13.1-py3-none-any.whl (11 kB)\n",
173
+ "Downloading multiprocess-0.70.16-py311-none-any.whl (143 kB)\n",
174
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.5/143.5 kB\u001b[0m \u001b[31m358.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
175
+ "\u001b[?25hDownloading networkx-3.2.1-py3-none-any.whl (1.6 MB)\n",
176
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m399.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
177
+ "\u001b[?25hDownloading pandas-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)\n",
178
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.0/13.0 MB\u001b[0m \u001b[31m221.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m \u001b[36m0:00:01\u001b[0m\n",
179
+ "\u001b[?25hDownloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)\n",
180
+ "Downloading scipy-1.12.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.4 MB)\n",
181
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m38.4/38.4 MB\u001b[0m \u001b[31m226.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
182
+ "\u001b[?25hDownloading sympy-1.12-py3-none-any.whl (5.7 MB)\n",
183
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.7/5.7 MB\u001b[0m \u001b[31m211.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
184
+ "\u001b[?25hDownloading xxhash-3.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
185
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.8/194.8 kB\u001b[0m \u001b[31m550.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
186
+ "\u001b[?25hDownloading mpmath-1.3.0-py3-none-any.whl (536 kB)\n",
187
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.2/536.2 kB\u001b[0m \u001b[31m560.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
188
+ "\u001b[?25hDownloading pytz-2024.1-py2.py3-none-any.whl (505 kB)\n",
189
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m505.5/505.5 kB\u001b[0m \u001b[31m554.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
190
+ "\u001b[?25hDownloading tzdata-2024.1-py2.py3-none-any.whl (345 kB)\n",
191
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m345.4/345.4 kB\u001b[0m \u001b[31m514.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
192
+ "\u001b[?25hDownloading nvidia_nvjitlink_cu12-12.3.101-py3-none-manylinux1_x86_64.whl (20.5 MB)\n",
193
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.5/20.5 MB\u001b[0m \u001b[31m243.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
194
+ "\u001b[?25hInstalling collected packages: pytz, mpmath, xxhash, tzdata, tqdm, sympy, scipy, safetensors, regex, pyarrow-hotfix, pyarrow, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, networkx, fsspec, filelock, einops, dill, triton, pandas, nvidia-cusparse-cu12, nvidia-cudnn-cu12, multiprocess, huggingface-hub, bitsandbytes, tokenizers, nvidia-cusolver-cu12, transformers, torch, datasets, accelerate, peft\n",
195
+ "Successfully installed accelerate-0.27.2 bitsandbytes-0.42.0 datasets-2.17.0 dill-0.3.8 einops-0.7.0 filelock-3.13.1 fsspec-2023.10.0 huggingface-hub-0.20.3 mpmath-1.3.0 multiprocess-0.70.16 networkx-3.2.1 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.3.101 nvidia-nvtx-cu12-12.1.105 pandas-2.2.0 peft-0.8.2 pyarrow-15.0.0 pyarrow-hotfix-0.6 pytz-2024.1 regex-2023.12.25 safetensors-0.4.2 scipy-1.12.0 sympy-1.12 tokenizers-0.15.2 torch-2.2.0 tqdm-4.66.2 transformers-4.37.2 triton-2.2.0 tzdata-2024.1 xxhash-3.4.1\n"
196
+ ]
197
+ }
198
+ ],
199
+ "source": [
200
+ "!pip install accelerate transformers einops datasets peft bitsandbytes torch"
201
+ ]
202
+ },
203
+ {
204
+ "cell_type": "code",
205
+ "execution_count": 2,
206
+ "id": "46a53303-b585-4b02-956f-4af173410e25",
207
+ "metadata": {},
208
+ "outputs": [
209
+ {
210
+ "name": "stderr",
211
+ "output_type": "stream",
212
+ "text": [
213
+ "/usr/local/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
214
+ " from .autonotebook import tqdm as notebook_tqdm\n"
215
+ ]
216
+ }
217
+ ],
218
+ "source": [
219
+ "import torch\n",
220
+ "from datasets import load_dataset, Dataset\n",
221
+ "from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer, DataCollatorForLanguageModeling, BitsAndBytesConfig\n",
222
+ "from peft import LoraConfig, get_peft_model\n",
223
+ "import os"
224
+ ]
225
+ },
226
+ {
227
+ "cell_type": "code",
228
+ "execution_count": 3,
229
+ "id": "ffa3ee11-3929-4032-8758-322cda3a912e",
230
+ "metadata": {},
231
+ "outputs": [
232
+ {
233
+ "name": "stderr",
234
+ "output_type": "stream",
235
+ "text": [
236
+ "tokenizer_config.json: 100%|██████████| 7.34k/7.34k [00:00<00:00, 35.6MB/s]\n",
237
+ "vocab.json: 100%|██████████| 798k/798k [00:00<00:00, 34.3MB/s]\n",
238
+ "merges.txt: 100%|██████████| 456k/456k [00:00<00:00, 22.9MB/s]\n",
239
+ "tokenizer.json: 100%|██████████| 2.11M/2.11M [00:00<00:00, 44.2MB/s]\n",
240
+ "added_tokens.json: 100%|██████████| 1.08k/1.08k [00:00<00:00, 9.74MB/s]\n",
241
+ "special_tokens_map.json: 100%|██████████| 99.0/99.0 [00:00<00:00, 882kB/s]\n",
242
+ "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
243
+ ]
244
+ }
245
+ ],
246
+ "source": [
247
+ "tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-2\", trust_remote_code=True)\n",
248
+ "tokenizer.pad_token = tokenizer.eos_token"
249
+ ]
250
+ },
251
+ {
252
+ "cell_type": "code",
253
+ "execution_count": 4,
254
+ "id": "b17096e6-c633-4d88-8cfc-fb738c1e4ca0",
255
+ "metadata": {
256
+ "scrolled": true
257
+ },
258
+ "outputs": [
259
+ {
260
+ "name": "stderr",
261
+ "output_type": "stream",
262
+ "text": [
263
+ "config.json: 100%|██████████| 863/863 [00:00<00:00, 7.14MB/s]\n",
264
+ "configuration_phi.py: 100%|██████████| 9.26k/9.26k [00:00<00:00, 28.2MB/s]\n",
265
+ "A new version of the following files was downloaded from https://huggingface.co/microsoft/phi-2:\n",
266
+ "- configuration_phi.py\n",
267
+ ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
268
+ "modeling_phi.py: 100%|██████████| 62.7k/62.7k [00:00<00:00, 157MB/s]\n",
269
+ "A new version of the following files was downloaded from https://huggingface.co/microsoft/phi-2:\n",
270
+ "- modeling_phi.py\n",
271
+ ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
272
+ "model.safetensors.index.json: 100%|██████████| 35.7k/35.7k [00:00<00:00, 118MB/s]\n",
273
+ "Downloading shards: 0%| | 0/2 [00:00<?, ?it/s]\n",
274
+ "model-00001-of-00002.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]\u001b[A\n",
275
+ "model-00001-of-00002.safetensors: 0%| | 21.0M/5.00G [00:00<00:24, 202MB/s]\u001b[A\n",
276
+ "model-00001-of-00002.safetensors: 1%|▏ | 62.9M/5.00G [00:00<00:16, 296MB/s]\u001b[A\n",
277
+ "model-00001-of-00002.safetensors: 2%|▏ | 94.4M/5.00G [00:01<01:49, 44.9MB/s]\u001b[A\n",
278
+ "model-00001-of-00002.safetensors: 2%|▏ | 115M/5.00G [00:03<02:49, 28.7MB/s] \u001b[A\n",
279
+ "model-00001-of-00002.safetensors: 3%|▎ | 136M/5.00G [00:03<02:07, 38.2MB/s]\u001b[A\n",
280
+ "model-00001-of-00002.safetensors: 4%|▎ | 178M/5.00G [00:03<01:14, 64.5MB/s]\u001b[A\n",
281
+ "model-00001-of-00002.safetensors: 4%|▍ | 199M/5.00G [00:04<02:15, 35.4MB/s]\u001b[A\n",
282
+ "model-00001-of-00002.safetensors: 4%|▍ | 220M/5.00G [00:05<02:53, 27.5MB/s]\u001b[A\n",
283
+ "model-00001-of-00002.safetensors: 5%|▍ | 241M/5.00G [00:06<02:14, 35.4MB/s]\u001b[A\n",
284
+ "model-00001-of-00002.safetensors: 5%|▌ | 273M/5.00G [00:06<01:29, 52.5MB/s]\u001b[A\n",
285
+ "model-00001-of-00002.safetensors: 6%|▋ | 315M/5.00G [00:06<00:57, 80.8MB/s]\u001b[A\n",
286
+ "model-00001-of-00002.safetensors: 7%|▋ | 357M/5.00G [00:06<00:41, 113MB/s] \u001b[A\n",
287
+ "model-00001-of-00002.safetensors: 8%|▊ | 388M/5.00G [00:07<01:29, 51.6MB/s]\u001b[A\n",
288
+ "model-00001-of-00002.safetensors: 8%|▊ | 419M/5.00G [00:09<01:58, 38.7MB/s]\u001b[A\n",
289
+ "model-00001-of-00002.safetensors: 9%|▉ | 440M/5.00G [00:09<01:38, 46.4MB/s]\u001b[A\n",
290
+ "model-00001-of-00002.safetensors: 9%|▉ | 472M/5.00G [00:09<01:11, 63.3MB/s]\u001b[A\n",
291
+ "model-00001-of-00002.safetensors: 10%|█ | 514M/5.00G [00:09<00:49, 91.3MB/s]\u001b[A\n",
292
+ "model-00001-of-00002.safetensors: 11%|█ | 545M/5.00G [00:10<01:31, 48.8MB/s]\u001b[A\n",
293
+ "model-00001-of-00002.safetensors: 11%|█▏ | 566M/5.00G [00:11<01:16, 57.5MB/s]\u001b[A\n",
294
+ "model-00001-of-00002.safetensors: 12%|█▏ | 587M/5.00G [00:12<02:07, 34.6MB/s]\u001b[A\n",
295
+ "model-00001-of-00002.safetensors: 12%|█▏ | 619M/5.00G [00:12<01:29, 48.7MB/s]\u001b[A\n",
296
+ "model-00001-of-00002.safetensors: 13%|█▎ | 661M/5.00G [00:12<00:59, 73.4MB/s]\u001b[A\n",
297
+ "model-00001-of-00002.safetensors: 14%|█▍ | 703M/5.00G [00:12<00:41, 103MB/s] \u001b[A\n",
298
+ "model-00001-of-00002.safetensors: 15%|█▍ | 744M/5.00G [00:12<00:31, 135MB/s]\u001b[A\n",
299
+ "model-00001-of-00002.safetensors: 16%|█▌ | 776M/5.00G [00:14<01:12, 58.4MB/s]\u001b[A\n",
300
+ "model-00001-of-00002.safetensors: 16%|█▌ | 807M/5.00G [00:14<00:57, 72.4MB/s]\u001b[A\n",
301
+ "model-00001-of-00002.safetensors: 17%|█▋ | 849M/5.00G [00:14<00:41, 99.3MB/s]\u001b[A\n",
302
+ "model-00001-of-00002.safetensors: 18%|█▊ | 891M/5.00G [00:14<00:31, 129MB/s] \u001b[A\n",
303
+ "model-00001-of-00002.safetensors: 19%|█▊ | 933M/5.00G [00:14<00:24, 163MB/s]\u001b[A\n",
304
+ "model-00001-of-00002.safetensors: 20%|█▉ | 975M/5.00G [00:14<00:20, 196MB/s]\u001b[A\n",
305
+ "model-00001-of-00002.safetensors: 20%|██ | 1.02G/5.00G [00:16<00:55, 71.7MB/s]\u001b[A\n",
306
+ "model-00001-of-00002.safetensors: 21%|██ | 1.05G/5.00G [00:16<00:46, 85.2MB/s]\u001b[A\n",
307
+ "model-00001-of-00002.safetensors: 22%|██▏ | 1.08G/5.00G [00:16<00:37, 105MB/s] \u001b[A\n",
308
+ "model-00001-of-00002.safetensors: 22%|██▏ | 1.12G/5.00G [00:16<00:28, 137MB/s]\u001b[A\n",
309
+ "model-00001-of-00002.safetensors: 23%|██▎ | 1.16G/5.00G [00:16<00:22, 167MB/s]\u001b[A\n",
310
+ "model-00001-of-00002.safetensors: 24%|██▍ | 1.20G/5.00G [00:16<00:20, 184MB/s]\u001b[A\n",
311
+ "model-00001-of-00002.safetensors: 25%|██▍ | 1.24G/5.00G [00:17<00:17, 218MB/s]\u001b[A\n",
312
+ "model-00001-of-00002.safetensors: 26%|██▌ | 1.28G/5.00G [00:17<00:15, 245MB/s]\u001b[A\n",
313
+ "model-00001-of-00002.safetensors: 26%|██▋ | 1.32G/5.00G [00:17<00:13, 271MB/s]\u001b[A\n",
314
+ "model-00001-of-00002.safetensors: 27%|██▋ | 1.36G/5.00G [00:17<00:12, 291MB/s]\u001b[A\n",
315
+ "model-00001-of-00002.safetensors: 28%|██▊ | 1.41G/5.00G [00:17<00:11, 307MB/s]\u001b[A\n",
316
+ "model-00001-of-00002.safetensors: 29%|██▉ | 1.45G/5.00G [00:17<00:11, 316MB/s]\u001b[A\n",
317
+ "model-00001-of-00002.safetensors: 30%|██▉ | 1.49G/5.00G [00:17<00:10, 327MB/s]\u001b[A\n",
318
+ "model-00001-of-00002.safetensors: 31%|███ | 1.53G/5.00G [00:17<00:10, 332MB/s]\u001b[A\n",
319
+ "model-00001-of-00002.safetensors: 31%|███▏ | 1.57G/5.00G [00:18<00:10, 337MB/s]\u001b[A\n",
320
+ "model-00001-of-00002.safetensors: 32%|███▏ | 1.61G/5.00G [00:18<00:09, 341MB/s]\u001b[A\n",
321
+ "model-00001-of-00002.safetensors: 33%|███▎ | 1.66G/5.00G [00:18<00:09, 338MB/s]\u001b[A\n",
322
+ "model-00001-of-00002.safetensors: 34%|███▍ | 1.70G/5.00G [00:18<00:09, 341MB/s]\u001b[A\n",
323
+ "model-00001-of-00002.safetensors: 35%|███▍ | 1.74G/5.00G [00:18<00:09, 347MB/s]\u001b[A\n",
324
+ "model-00001-of-00002.safetensors: 36%|███▌ | 1.78G/5.00G [00:18<00:09, 339MB/s]\u001b[A\n",
325
+ "model-00001-of-00002.safetensors: 37%|███▋ | 1.82G/5.00G [00:18<00:09, 340MB/s]\u001b[A\n",
326
+ "model-00001-of-00002.safetensors: 37%|███▋ | 1.87G/5.00G [00:18<00:09, 338MB/s]\u001b[A\n",
327
+ "model-00001-of-00002.safetensors: 38%|███▊ | 1.91G/5.00G [00:19<00:09, 342MB/s]\u001b[A\n",
328
+ "model-00001-of-00002.safetensors: 39%|███▉ | 1.95G/5.00G [00:19<00:08, 344MB/s]\u001b[A\n",
329
+ "model-00001-of-00002.safetensors: 40%|███▉ | 1.99G/5.00G [00:19<00:08, 340MB/s]\u001b[A\n",
330
+ "model-00001-of-00002.safetensors: 41%|████ | 2.03G/5.00G [00:19<00:08, 345MB/s]\u001b[A\n",
331
+ "model-00001-of-00002.safetensors: 42%|████▏ | 2.08G/5.00G [00:19<00:08, 343MB/s]\u001b[A\n",
332
+ "model-00001-of-00002.safetensors: 42%|████▏ | 2.12G/5.00G [00:19<00:08, 340MB/s]\u001b[A\n",
333
+ "model-00001-of-00002.safetensors: 43%|████▎ | 2.16G/5.00G [00:19<00:08, 341MB/s]\u001b[A\n",
334
+ "model-00001-of-00002.safetensors: 44%|████▍ | 2.20G/5.00G [00:19<00:08, 324MB/s]\u001b[A\n",
335
+ "model-00001-of-00002.safetensors: 45%|████▍ | 2.24G/5.00G [00:20<00:08, 328MB/s]\u001b[A\n",
336
+ "model-00001-of-00002.safetensors: 46%|████▌ | 2.29G/5.00G [00:20<00:08, 335MB/s]\u001b[A\n",
337
+ "model-00001-of-00002.safetensors: 47%|████▋ | 2.33G/5.00G [00:20<00:07, 336MB/s]\u001b[A\n",
338
+ "model-00001-of-00002.safetensors: 47%|████▋ | 2.37G/5.00G [00:20<00:07, 339MB/s]\u001b[A\n",
339
+ "model-00001-of-00002.safetensors: 48%|████▊ | 2.41G/5.00G [00:20<00:07, 345MB/s]\u001b[A\n",
340
+ "model-00001-of-00002.safetensors: 49%|████▉ | 2.45G/5.00G [00:20<00:07, 340MB/s]\u001b[A\n",
341
+ "model-00001-of-00002.safetensors: 50%|████▉ | 2.50G/5.00G [00:20<00:07, 329MB/s]\u001b[A\n",
342
+ "model-00001-of-00002.safetensors: 51%|█████ | 2.54G/5.00G [00:20<00:08, 299MB/s]\u001b[A\n",
343
+ "model-00001-of-00002.safetensors: 51%|█████▏ | 2.57G/5.00G [00:21<00:08, 292MB/s]\u001b[A\n",
344
+ "model-00001-of-00002.safetensors: 52%|█████▏ | 2.61G/5.00G [00:21<00:07, 307MB/s]\u001b[A\n",
345
+ "model-00001-of-00002.safetensors: 53%|█████▎ | 2.65G/5.00G [00:21<00:07, 322MB/s]\u001b[A\n",
346
+ "model-00001-of-00002.safetensors: 54%|█████▍ | 2.69G/5.00G [00:21<00:06, 331MB/s]\u001b[A\n",
347
+ "model-00001-of-00002.safetensors: 55%|█████▍ | 2.74G/5.00G [00:21<00:06, 337MB/s]\u001b[A\n",
348
+ "model-00001-of-00002.safetensors: 56%|█████▌ | 2.78G/5.00G [00:21<00:06, 344MB/s]\u001b[A\n",
349
+ "model-00001-of-00002.safetensors: 56%|█████▋ | 2.82G/5.00G [00:21<00:06, 329MB/s]\u001b[A\n",
350
+ "model-00001-of-00002.safetensors: 57%|█████▋ | 2.86G/5.00G [00:21<00:06, 335MB/s]\u001b[A\n",
351
+ "model-00001-of-00002.safetensors: 58%|█████▊ | 2.90G/5.00G [00:22<00:06, 335MB/s]\u001b[A\n",
352
+ "model-00001-of-00002.safetensors: 59%|█████▉ | 2.95G/5.00G [00:22<00:06, 339MB/s]\u001b[A\n",
353
+ "model-00001-of-00002.safetensors: 60%|█████▉ | 2.99G/5.00G [00:22<00:06, 334MB/s]\u001b[A\n",
354
+ "model-00001-of-00002.safetensors: 61%|██████ | 3.03G/5.00G [00:22<00:05, 340MB/s]\u001b[A\n",
355
+ "model-00001-of-00002.safetensors: 62%|██████▏ | 3.07G/5.00G [00:22<00:05, 341MB/s]\u001b[A\n",
356
+ "model-00001-of-00002.safetensors: 62%|██████▏ | 3.11G/5.00G [00:22<00:05, 340MB/s]\u001b[A\n",
357
+ "model-00001-of-00002.safetensors: 63%|██████▎ | 3.16G/5.00G [00:22<00:05, 345MB/s]\u001b[A\n",
358
+ "model-00001-of-00002.safetensors: 64%|██████▍ | 3.20G/5.00G [00:22<00:05, 342MB/s]\u001b[A\n",
359
+ "model-00001-of-00002.safetensors: 65%|██████▍ | 3.24G/5.00G [00:23<00:05, 339MB/s]\u001b[A\n",
360
+ "model-00001-of-00002.safetensors: 66%|██████▌ | 3.28G/5.00G [00:23<00:05, 322MB/s]\u001b[A\n",
361
+ "model-00001-of-00002.safetensors: 67%|██████▋ | 3.32G/5.00G [00:23<00:05, 329MB/s]\u001b[A\n",
362
+ "model-00001-of-00002.safetensors: 67%|██████▋ | 3.37G/5.00G [00:23<00:04, 334MB/s]\u001b[A\n",
363
+ "model-00001-of-00002.safetensors: 68%|██████▊ | 3.41G/5.00G [00:23<00:04, 335MB/s]\u001b[A\n",
364
+ "model-00001-of-00002.safetensors: 69%|██████▉ | 3.45G/5.00G [00:23<00:04, 342MB/s]\u001b[A\n",
365
+ "model-00001-of-00002.safetensors: 70%|██████▉ | 3.49G/5.00G [00:23<00:04, 347MB/s]\u001b[A\n",
366
+ "model-00001-of-00002.safetensors: 71%|███████ | 3.53G/5.00G [00:23<00:04, 349MB/s]\u001b[A\n",
367
+ "model-00001-of-00002.safetensors: 72%|███████▏ | 3.58G/5.00G [00:24<00:04, 349MB/s]\u001b[A\n",
368
+ "model-00001-of-00002.safetensors: 72%|███████▏ | 3.62G/5.00G [00:24<00:03, 350MB/s]\u001b[A\n",
369
+ "model-00001-of-00002.safetensors: 73%|███████▎ | 3.66G/5.00G [00:24<00:03, 346MB/s]\u001b[A\n",
370
+ "model-00001-of-00002.safetensors: 74%|███████▍ | 3.70G/5.00G [00:24<00:03, 334MB/s]\u001b[A\n",
371
+ "model-00001-of-00002.safetensors: 75%|███████▍ | 3.74G/5.00G [00:24<00:03, 339MB/s]\u001b[A\n",
372
+ "model-00001-of-00002.safetensors: 76%|███████▌ | 3.79G/5.00G [00:24<00:03, 340MB/s]\u001b[A\n",
373
+ "model-00001-of-00002.safetensors: 77%|███████▋ | 3.83G/5.00G [00:24<00:03, 344MB/s]\u001b[A\n",
374
+ "model-00001-of-00002.safetensors: 77%|███████▋ | 3.87G/5.00G [00:24<00:03, 343MB/s]\u001b[A\n",
375
+ "model-00001-of-00002.safetensors: 78%|███████▊ | 3.91G/5.00G [00:25<00:03, 337MB/s]\u001b[A\n",
376
+ "model-00001-of-00002.safetensors: 79%|███████▉ | 3.95G/5.00G [00:25<00:03, 342MB/s]\u001b[A\n",
377
+ "model-00001-of-00002.safetensors: 80%|███████▉ | 4.00G/5.00G [00:25<00:02, 346MB/s]\u001b[A\n",
378
+ "model-00001-of-00002.safetensors: 81%|████████ | 4.04G/5.00G [00:25<00:02, 341MB/s]\u001b[A\n",
379
+ "model-00001-of-00002.safetensors: 82%|████████▏ | 4.08G/5.00G [00:25<00:02, 347MB/s]\u001b[A\n",
380
+ "model-00001-of-00002.safetensors: 82%|████████▏ | 4.12G/5.00G [00:25<00:02, 353MB/s]\u001b[A\n",
381
+ "model-00001-of-00002.safetensors: 83%|████████▎ | 4.16G/5.00G [00:25<00:02, 351MB/s]\u001b[A\n",
382
+ "model-00001-of-00002.safetensors: 84%|████████▍ | 4.20G/5.00G [00:25<00:02, 350MB/s]\u001b[A\n",
383
+ "model-00001-of-00002.safetensors: 85%|████████��� | 4.25G/5.00G [00:25<00:02, 351MB/s]\u001b[A\n",
384
+ "model-00001-of-00002.safetensors: 86%|████████▌ | 4.29G/5.00G [00:26<00:02, 348MB/s]\u001b[A\n",
385
+ "model-00001-of-00002.safetensors: 87%|████████▋ | 4.33G/5.00G [00:26<00:01, 349MB/s]\u001b[A\n",
386
+ "model-00001-of-00002.safetensors: 88%|████████▊ | 4.37G/5.00G [00:26<00:01, 335MB/s]\u001b[A\n",
387
+ "model-00001-of-00002.safetensors: 88%|████████▊ | 4.41G/5.00G [00:26<00:01, 332MB/s]\u001b[A\n",
388
+ "model-00001-of-00002.safetensors: 89%|████████▉ | 4.46G/5.00G [00:26<00:01, 337MB/s]\u001b[A\n",
389
+ "model-00001-of-00002.safetensors: 90%|█████████ | 4.50G/5.00G [00:26<00:01, 305MB/s]\u001b[A\n",
390
+ "model-00001-of-00002.safetensors: 91%|█████████ | 4.54G/5.00G [00:26<00:01, 318MB/s]\u001b[A\n",
391
+ "model-00001-of-00002.safetensors: 92%|█████████▏| 4.58G/5.00G [00:27<00:01, 318MB/s]\u001b[A\n",
392
+ "model-00001-of-00002.safetensors: 93%|█████████▎| 4.62G/5.00G [00:27<00:01, 281MB/s]\u001b[A\n",
393
+ "model-00001-of-00002.safetensors: 93%|█████████▎| 4.66G/5.00G [00:27<00:01, 286MB/s]\u001b[A\n",
394
+ "model-00001-of-00002.safetensors: 94%|█████████▍| 4.70G/5.00G [00:27<00:00, 303MB/s]\u001b[A\n",
395
+ "model-00001-of-00002.safetensors: 95%|█████████▍| 4.73G/5.00G [00:27<00:01, 246MB/s]\u001b[A\n",
396
+ "model-00001-of-00002.safetensors: 95%|█████████▌| 4.76G/5.00G [00:27<00:00, 252MB/s]\u001b[A\n",
397
+ "model-00001-of-00002.safetensors: 96%|█████████▌| 4.79G/5.00G [00:27<00:00, 254MB/s]\u001b[A\n",
398
+ "model-00001-of-00002.safetensors: 97%|█████████▋| 4.83G/5.00G [00:27<00:00, 277MB/s]\u001b[A\n",
399
+ "model-00001-of-00002.safetensors: 98%|█████████▊| 4.88G/5.00G [00:28<00:00, 296MB/s]\u001b[A\n",
400
+ "model-00001-of-00002.safetensors: 98%|█████████▊| 4.92G/5.00G [00:28<00:00, 309MB/s]\u001b[A\n",
401
+ "model-00001-of-00002.safetensors: 99%|█████████▉| 4.96G/5.00G [00:28<00:00, 320MB/s]\u001b[A\n",
402
+ "model-00001-of-00002.safetensors: 100%|██████████| 5.00G/5.00G [00:28<00:00, 176MB/s]\u001b[A\n",
403
+ "Downloading shards: 50%|█████ | 1/2 [00:28<00:28, 28.60s/it]\n",
404
+ "model-00002-of-00002.safetensors: 0%| | 0.00/564M [00:00<?, ?B/s]\u001b[A\n",
405
+ "model-00002-of-00002.safetensors: 7%|▋ | 41.9M/564M [00:00<00:01, 357MB/s]\u001b[A\n",
406
+ "model-00002-of-00002.safetensors: 15%|█▍ | 83.9M/564M [00:00<00:01, 336MB/s]\u001b[A\n",
407
+ "model-00002-of-00002.safetensors: 22%|██▏ | 126M/564M [00:00<00:01, 345MB/s] \u001b[A\n",
408
+ "model-00002-of-00002.safetensors: 30%|██▉ | 168M/564M [00:00<00:01, 349MB/s]\u001b[A\n",
409
+ "model-00002-of-00002.safetensors: 37%|███▋ | 210M/564M [00:00<00:01, 348MB/s]\u001b[A\n",
410
+ "model-00002-of-00002.safetensors: 45%|████▍ | 252M/564M [00:00<00:00, 323MB/s]\u001b[A\n",
411
+ "model-00002-of-00002.safetensors: 52%|█████▏ | 294M/564M [00:00<00:00, 324MB/s]\u001b[A\n",
412
+ "model-00002-of-00002.safetensors: 60%|█████▉ | 336M/564M [00:01<00:00, 320MB/s]\u001b[A\n",
413
+ "model-00002-of-00002.safetensors: 67%|██████▋ | 377M/564M [00:01<00:00, 321MB/s]\u001b[A\n",
414
+ "model-00002-of-00002.safetensors: 74%|███████▍ | 419M/564M [00:01<00:00, 331MB/s]\u001b[A\n",
415
+ "model-00002-of-00002.safetensors: 82%|████████▏ | 461M/564M [00:01<00:00, 327MB/s]\u001b[A\n",
416
+ "model-00002-of-00002.safetensors: 89%|████████▉ | 503M/564M [00:01<00:00, 331MB/s]\u001b[A\n",
417
+ "model-00002-of-00002.safetensors: 100%|██████████| 564M/564M [00:01<00:00, 333MB/s]\u001b[A\n",
418
+ "Downloading shards: 100%|██████████| 2/2 [00:30<00:00, 15.18s/it]\n",
419
+ "Loading checkpoint shards: 100%|██████████| 2/2 [00:03<00:00, 1.55s/it]\n",
420
+ "generation_config.json: 100%|██████████| 124/124 [00:00<00:00, 987kB/s]\n"
421
+ ]
422
+ }
423
+ ],
424
+ "source": [
425
+ "bnb_config = BitsAndBytesConfig(\n",
426
+ " load_in_4bit=True,\n",
427
+ " bnb_4bit_use_double_quant=True,\n",
428
+ " bnb_4bit_quant_type=\"nf4\",\n",
429
+ " bnb_4bit_compute_dtype=torch.float16\n",
430
+ ")\n",
431
+ "\n",
432
+ "model = AutoModelForCausalLM.from_pretrained(\n",
433
+ " \"microsoft/phi-2\",\n",
434
+ " device_map={\"\":0},\n",
435
+ " trust_remote_code=True,\n",
436
+ " quantization_config=bnb_config\n",
437
+ ")"
438
+ ]
439
+ },
440
+ {
441
+ "cell_type": "code",
442
+ "execution_count": 5,
443
+ "id": "e15aa794-e17c-4b09-a64a-c60377259218",
444
+ "metadata": {},
445
+ "outputs": [
446
+ {
447
+ "name": "stdout",
448
+ "output_type": "stream",
449
+ "text": [
450
+ "PhiForCausalLM(\n",
451
+ " (model): PhiModel(\n",
452
+ " (embed_tokens): Embedding(51200, 2560)\n",
453
+ " (embed_dropout): Dropout(p=0.0, inplace=False)\n",
454
+ " (layers): ModuleList(\n",
455
+ " (0-31): 32 x PhiDecoderLayer(\n",
456
+ " (self_attn): PhiAttention(\n",
457
+ " (q_proj): Linear4bit(in_features=2560, out_features=2560, bias=True)\n",
458
+ " (k_proj): Linear4bit(in_features=2560, out_features=2560, bias=True)\n",
459
+ " (v_proj): Linear4bit(in_features=2560, out_features=2560, bias=True)\n",
460
+ " (dense): Linear4bit(in_features=2560, out_features=2560, bias=True)\n",
461
+ " (rotary_emb): PhiRotaryEmbedding()\n",
462
+ " )\n",
463
+ " (mlp): PhiMLP(\n",
464
+ " (activation_fn): NewGELUActivation()\n",
465
+ " (fc1): Linear4bit(in_features=2560, out_features=10240, bias=True)\n",
466
+ " (fc2): Linear4bit(in_features=10240, out_features=2560, bias=True)\n",
467
+ " )\n",
468
+ " (input_layernorm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)\n",
469
+ " (resid_dropout): Dropout(p=0.1, inplace=False)\n",
470
+ " )\n",
471
+ " )\n",
472
+ " (final_layernorm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)\n",
473
+ " )\n",
474
+ " (lm_head): Linear(in_features=2560, out_features=51200, bias=True)\n",
475
+ ")\n"
476
+ ]
477
+ }
478
+ ],
479
+ "source": [
480
+ "print(model)"
481
+ ]
482
+ },
483
+ {
484
+ "cell_type": "code",
485
+ "execution_count": 20,
486
+ "id": "18d5599f-992d-4d8e-a90c-4d43774be473",
487
+ "metadata": {},
488
+ "outputs": [
489
+ {
490
+ "name": "stdout",
491
+ "output_type": "stream",
492
+ "text": [
493
+ "trainable params: 18,350,080 || all params: 2,798,033,920 || trainable%: 0.6558204984162593\n"
494
+ ]
495
+ }
496
+ ],
497
+ "source": [
498
+ "config = LoraConfig(\n",
499
+ " r=16,\n",
500
+ " lora_alpha=16,\n",
501
+ " #target_modules=[\"q_proj\",\"k_proj\",\"v_proj\"],\n",
502
+ " lora_dropout=0.05,\n",
503
+ " bias=\"none\",\n",
504
+ " task_type=\"CAUSAL_LM\"\n",
505
+ ")\n",
506
+ "\n",
507
+ "model = get_peft_model(model, config)\n",
508
+ "model.print_trainable_parameters()"
509
+ ]
510
+ },
511
+ {
512
+ "cell_type": "code",
513
+ "execution_count": 7,
514
+ "id": "baeee903-3dce-48b2-93c3-7a697d8c6daf",
515
+ "metadata": {},
516
+ "outputs": [],
517
+ "source": [
518
+ "def tokenize(sample):\n",
519
+ " model_inps = tokenizer(sample[\"text\"], padding=True, truncation=True, max_length=512)\n",
520
+ " return model_inps"
521
+ ]
522
+ },
523
+ {
524
+ "cell_type": "code",
525
+ "execution_count": 8,
526
+ "id": "28a9b24a-a822-4fcb-96b3-d77b7ea30a5f",
527
+ "metadata": {},
528
+ "outputs": [
529
+ {
530
+ "name": "stdout",
531
+ "output_type": "stream",
532
+ "text": [
533
+ "Collecting scikit-learn\n",
534
+ " Downloading scikit_learn-1.4.1.post1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)\n",
535
+ "Requirement already satisfied: numpy<2.0,>=1.19.5 in /usr/local/lib/python3.11/site-packages (from scikit-learn) (1.26.4)\n",
536
+ "Requirement already satisfied: scipy>=1.6.0 in /usr/local/lib/python3.11/site-packages (from scikit-learn) (1.12.0)\n",
537
+ "Collecting joblib>=1.2.0 (from scikit-learn)\n",
538
+ " Downloading joblib-1.3.2-py3-none-any.whl.metadata (5.4 kB)\n",
539
+ "Collecting threadpoolctl>=2.0.0 (from scikit-learn)\n",
540
+ " Downloading threadpoolctl-3.3.0-py3-none-any.whl.metadata (13 kB)\n",
541
+ "Downloading scikit_learn-1.4.1.post1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB)\n",
542
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.1/12.1 MB\u001b[0m \u001b[31m183.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m \u001b[36m0:00:01\u001b[0m\n",
543
+ "\u001b[?25hDownloading joblib-1.3.2-py3-none-any.whl (302 kB)\n",
544
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.2/302.2 kB\u001b[0m \u001b[31m531.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
545
+ "\u001b[?25hDownloading threadpoolctl-3.3.0-py3-none-any.whl (17 kB)\n",
546
+ "Installing collected packages: threadpoolctl, joblib, scikit-learn\n",
547
+ "Successfully installed joblib-1.3.2 scikit-learn-1.4.1.post1 threadpoolctl-3.3.0\n"
548
+ ]
549
+ }
550
+ ],
551
+ "source": [
552
+ "!pip install scikit-learn"
553
+ ]
554
+ },
555
+ {
556
+ "cell_type": "code",
557
+ "execution_count": 9,
558
+ "id": "1ee7fd2a-38e4-4f23-a978-0bdeeda64d8b",
559
+ "metadata": {},
560
+ "outputs": [],
561
+ "source": [
562
+ "import pandas as pd\n",
563
+ "\n",
564
+ "from sklearn.model_selection import train_test_split\n",
565
+ "dataset_name='data.csv'\n",
566
+ "df = pd.read_csv(dataset_name)\n",
567
+ "#train, test = train_test_split(df, test_size=0.2)"
568
+ ]
569
+ },
570
+ {
571
+ "cell_type": "code",
572
+ "execution_count": 10,
573
+ "id": "e84c29e2-843e-42c2-8c0f-324d392e671c",
574
+ "metadata": {},
575
+ "outputs": [
576
+ {
577
+ "name": "stderr",
578
+ "output_type": "stream",
579
+ "text": [
580
+ "Tokenizing data: 100%|██████████| 16412/16412 [00:02<00:00, 6676.76 examples/s]\n"
581
+ ]
582
+ },
583
+ {
584
+ "data": {
585
+ "text/plain": [
586
+ "Dataset({\n",
587
+ " features: ['input_ids', 'attention_mask'],\n",
588
+ " num_rows: 16412\n",
589
+ "})"
590
+ ]
591
+ },
592
+ "execution_count": 10,
593
+ "metadata": {},
594
+ "output_type": "execute_result"
595
+ }
596
+ ],
597
+ "source": [
598
+ "data_df = df\n",
599
+ "data_df[\"text\"] = data_df[[\"user\", \"assistant\"]].apply(lambda x: \"question: \" + str(x[\"user\"]) + \" answer: \" + str(x[\"assistant\"]), axis=1)\n",
600
+ "data = Dataset.from_pandas(data_df)\n",
601
+ "tokenized_data = data.map(tokenize, batched=True, desc=\"Tokenizing data\", remove_columns=data.column_names)\n",
602
+ "tokenized_data"
603
+ ]
604
+ },
605
+ {
606
+ "cell_type": "code",
607
+ "execution_count": 21,
608
+ "id": "ac968254-5338-49df-950d-222b82647407",
609
+ "metadata": {},
610
+ "outputs": [],
611
+ "source": [
612
+ "training_arguments = TrainingArguments(\n",
613
+ " output_dir=\".\",\n",
614
+ " per_device_train_batch_size=4,\n",
615
+ " gradient_accumulation_steps=1,\n",
616
+ " learning_rate=2e-4,\n",
617
+ " lr_scheduler_type=\"cosine\",\n",
618
+ " save_strategy=\"epoch\",\n",
619
+ " logging_steps=100,\n",
620
+ " max_steps=2000,\n",
621
+ " num_train_epochs=3,\n",
622
+ " push_to_hub=True\n",
623
+ " )"
624
+ ]
625
+ },
626
+ {
627
+ "cell_type": "code",
628
+ "execution_count": 12,
629
+ "id": "729df2d0-0890-4ac4-adf3-c167a6e9669d",
630
+ "metadata": {},
631
+ "outputs": [
632
+ {
633
+ "name": "stderr",
634
+ "output_type": "stream",
635
+ "text": [
636
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
637
+ "To disable this warning, you can either:\n",
638
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
639
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
640
+ ]
641
+ },
642
+ {
643
+ "name": "stdout",
644
+ "output_type": "stream",
645
+ "text": [
646
+ "Requirement already satisfied: huggingface_hub in /usr/local/lib/python3.11/site-packages (0.20.3)\n",
647
+ "Requirement already satisfied: filelock in /usr/local/lib/python3.11/site-packages (from huggingface_hub) (3.13.1)\n",
648
+ "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.11/site-packages (from huggingface_hub) (2023.10.0)\n",
649
+ "Requirement already satisfied: requests in /usr/local/lib/python3.11/site-packages (from huggingface_hub) (2.31.0)\n",
650
+ "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.11/site-packages (from huggingface_hub) (4.66.2)\n",
651
+ "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.11/site-packages (from huggingface_hub) (6.0.1)\n",
652
+ "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.11/site-packages (from huggingface_hub) (4.9.0)\n",
653
+ "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.11/site-packages (from huggingface_hub) (23.2)\n",
654
+ "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/site-packages (from requests->huggingface_hub) (2.1.1)\n",
655
+ "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/site-packages (from requests->huggingface_hub) (3.6)\n",
656
+ "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/site-packages (from requests->huggingface_hub) (2.2.0)\n",
657
+ "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/site-packages (from requests->huggingface_hub) (2024.2.2)\n"
658
+ ]
659
+ }
660
+ ],
661
+ "source": [
662
+ "!pip install huggingface_hub"
663
+ ]
664
+ },
665
+ {
666
+ "cell_type": "code",
667
+ "execution_count": 13,
668
+ "id": "525f81a4-eb92-466e-bb9c-cd63122231ab",
669
+ "metadata": {},
670
+ "outputs": [
671
+ {
672
+ "name": "stderr",
673
+ "output_type": "stream",
674
+ "text": [
675
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
676
+ "To disable this warning, you can either:\n",
677
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
678
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
679
+ ]
680
+ },
681
+ {
682
+ "name": "stdout",
683
+ "output_type": "stream",
684
+ "text": [
685
+ "Requirement already satisfied: jupyterlab in /usr/local/lib/python3.11/site-packages (4.1.0)\n",
686
+ "Collecting jupyterlab\n",
687
+ " Downloading jupyterlab-4.1.1-py3-none-any.whl.metadata (15 kB)\n",
688
+ "Collecting jupyterlab-git\n",
689
+ " Downloading jupyterlab_git-0.50.0-py3-none-any.whl.metadata (31 kB)\n",
690
+ "Requirement already satisfied: async-lru>=1.0.0 in /usr/local/lib/python3.11/site-packages (from jupyterlab) (2.0.4)\n",
691
+ "Requirement already satisfied: httpx>=0.25.0 in /usr/local/lib/python3.11/site-packages (from jupyterlab) (0.26.0)\n",
692
+ "Requirement already satisfied: ipykernel in /usr/local/lib/python3.11/site-packages (from jupyterlab) (6.29.1)\n",
693
+ "Requirement already satisfied: jinja2>=3.0.3 in /usr/local/lib/python3.11/site-packages (from jupyterlab) (3.1.3)\n",
694
+ "Requirement already satisfied: jupyter-core in /usr/local/lib/python3.11/site-packages (from jupyterlab) (5.7.1)\n",
695
+ "Requirement already satisfied: jupyter-lsp>=2.0.0 in /usr/local/lib/python3.11/site-packages (from jupyterlab) (2.2.2)\n",
696
+ "Requirement already satisfied: jupyter-server<3,>=2.4.0 in /usr/local/lib/python3.11/site-packages (from jupyterlab) (2.12.5)\n",
697
+ "Requirement already satisfied: jupyterlab-server<3,>=2.19.0 in /usr/local/lib/python3.11/site-packages (from jupyterlab) (2.25.2)\n",
698
+ "Requirement already satisfied: notebook-shim>=0.2 in /usr/local/lib/python3.11/site-packages (from jupyterlab) (0.2.3)\n",
699
+ "Requirement already satisfied: packaging in /usr/local/lib/python3.11/site-packages (from jupyterlab) (23.2)\n",
700
+ "Requirement already satisfied: tornado>=6.2.0 in /usr/local/lib/python3.11/site-packages (from jupyterlab) (6.4)\n",
701
+ "Requirement already satisfied: traitlets in /usr/local/lib/python3.11/site-packages (from jupyterlab) (5.14.1)\n",
702
+ "Collecting nbdime~=4.0.1 (from jupyterlab-git)\n",
703
+ " Downloading nbdime-4.0.1-py3-none-any.whl.metadata (9.5 kB)\n",
704
+ "Requirement already satisfied: nbformat in /usr/local/lib/python3.11/site-packages (from jupyterlab-git) (5.9.2)\n",
705
+ "Requirement already satisfied: pexpect in /usr/local/lib/python3.11/site-packages (from jupyterlab-git) (4.9.0)\n",
706
+ "Requirement already satisfied: anyio in /usr/local/lib/python3.11/site-packages (from httpx>=0.25.0->jupyterlab) (4.2.0)\n",
707
+ "Requirement already satisfied: certifi in /usr/local/lib/python3.11/site-packages (from httpx>=0.25.0->jupyterlab) (2024.2.2)\n",
708
+ "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.11/site-packages (from httpx>=0.25.0->jupyterlab) (1.0.2)\n",
709
+ "Requirement already satisfied: idna in /usr/local/lib/python3.11/site-packages (from httpx>=0.25.0->jupyterlab) (3.6)\n",
710
+ "Requirement already satisfied: sniffio in /usr/local/lib/python3.11/site-packages (from httpx>=0.25.0->jupyterlab) (1.3.0)\n",
711
+ "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.11/site-packages (from httpcore==1.*->httpx>=0.25.0->jupyterlab) (0.14.0)\n",
712
+ "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/site-packages (from jinja2>=3.0.3->jupyterlab) (2.1.5)\n",
713
+ "Requirement already satisfied: argon2-cffi in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (23.1.0)\n",
714
+ "Requirement already satisfied: jupyter-client>=7.4.4 in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (8.6.0)\n",
715
+ "Requirement already satisfied: jupyter-events>=0.9.0 in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (0.9.0)\n",
716
+ "Requirement already satisfied: jupyter-server-terminals in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (0.5.2)\n",
717
+ "Requirement already satisfied: nbconvert>=6.4.4 in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (7.15.0)\n",
718
+ "Requirement already satisfied: overrides in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (7.7.0)\n",
719
+ "Requirement already satisfied: prometheus-client in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (0.19.0)\n",
720
+ "Requirement already satisfied: pyzmq>=24 in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (25.1.2)\n",
721
+ "Requirement already satisfied: send2trash>=1.8.2 in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (1.8.2)\n",
722
+ "Requirement already satisfied: terminado>=0.8.3 in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (0.18.0)\n",
723
+ "Requirement already satisfied: websocket-client in /usr/local/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab) (1.7.0)\n",
724
+ "Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.11/site-packages (from jupyter-core->jupyterlab) (4.2.0)\n",
725
+ "Requirement already satisfied: babel>=2.10 in /usr/local/lib/python3.11/site-packages (from jupyterlab-server<3,>=2.19.0->jupyterlab) (2.14.0)\n",
726
+ "Requirement already satisfied: json5>=0.9.0 in /usr/local/lib/python3.11/site-packages (from jupyterlab-server<3,>=2.19.0->jupyterlab) (0.9.14)\n",
727
+ "Requirement already satisfied: jsonschema>=4.18.0 in /usr/local/lib/python3.11/site-packages (from jupyterlab-server<3,>=2.19.0->jupyterlab) (4.21.1)\n",
728
+ "Requirement already satisfied: requests>=2.31 in /usr/local/lib/python3.11/site-packages (from jupyterlab-server<3,>=2.19.0->jupyterlab) (2.31.0)\n",
729
+ "Collecting colorama (from nbdime~=4.0.1->jupyterlab-git)\n",
730
+ " Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
731
+ "Collecting gitpython!=2.1.4,!=2.1.5,!=2.1.6 (from nbdime~=4.0.1->jupyterlab-git)\n",
732
+ " Downloading GitPython-3.1.42-py3-none-any.whl.metadata (12 kB)\n",
733
+ "Collecting jupyter-server-mathjax>=0.2.2 (from nbdime~=4.0.1->jupyterlab-git)\n",
734
+ " Downloading jupyter_server_mathjax-0.2.6-py3-none-any.whl (3.1 MB)\n",
735
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m110.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
736
+ "\u001b[?25hRequirement already satisfied: pygments in /usr/local/lib/python3.11/site-packages (from nbdime~=4.0.1->jupyterlab-git) (2.17.2)\n",
737
+ "Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.11/site-packages (from nbformat->jupyterlab-git) (2.19.1)\n",
738
+ "Requirement already satisfied: comm>=0.1.1 in /usr/local/lib/python3.11/site-packages (from ipykernel->jupyterlab) (0.2.1)\n",
739
+ "Requirement already satisfied: debugpy>=1.6.5 in /usr/local/lib/python3.11/site-packages (from ipykernel->jupyterlab) (1.8.0)\n",
740
+ "Requirement already satisfied: ipython>=7.23.1 in /usr/local/lib/python3.11/site-packages (from ipykernel->jupyterlab) (8.21.0)\n",
741
+ "Requirement already satisfied: matplotlib-inline>=0.1 in /usr/local/lib/python3.11/site-packages (from ipykernel->jupyterlab) (0.1.6)\n",
742
+ "Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.11/site-packages (from ipykernel->jupyterlab) (1.6.0)\n",
743
+ "Requirement already satisfied: psutil in /usr/local/lib/python3.11/site-packages (from ipykernel->jupyterlab) (5.9.8)\n",
744
+ "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.11/site-packages (from pexpect->jupyterlab-git) (0.7.0)\n",
745
+ "Collecting gitdb<5,>=4.0.1 (from gitpython!=2.1.4,!=2.1.5,!=2.1.6->nbdime~=4.0.1->jupyterlab-git)\n",
746
+ " Downloading gitdb-4.0.11-py3-none-any.whl.metadata (1.2 kB)\n",
747
+ "Requirement already satisfied: decorator in /usr/local/lib/python3.11/site-packages (from ipython>=7.23.1->ipykernel->jupyterlab) (5.1.1)\n",
748
+ "Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.11/site-packages (from ipython>=7.23.1->ipykernel->jupyterlab) (0.19.1)\n",
749
+ "Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /usr/local/lib/python3.11/site-packages (from ipython>=7.23.1->ipykernel->jupyterlab) (3.0.43)\n",
750
+ "Requirement already satisfied: stack-data in /usr/local/lib/python3.11/site-packages (from ipython>=7.23.1->ipykernel->jupyterlab) (0.6.3)\n",
751
+ "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.11/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.19.0->jupyterlab) (23.2.0)\n",
752
+ "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.11/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.19.0->jupyterlab) (2023.12.1)\n",
753
+ "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.11/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.19.0->jupyterlab) (0.33.0)\n",
754
+ "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.11/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.19.0->jupyterlab) (0.17.1)\n",
755
+ "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/site-packages (from jupyter-client>=7.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (2.8.2)\n",
756
+ "Requirement already satisfied: python-json-logger>=2.0.4 in /usr/local/lib/python3.11/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (2.0.7)\n",
757
+ "Requirement already satisfied: pyyaml>=5.3 in /usr/local/lib/python3.11/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (6.0.1)\n",
758
+ "Requirement already satisfied: rfc3339-validator in /usr/local/lib/python3.11/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (0.1.4)\n",
759
+ "Requirement already satisfied: rfc3986-validator>=0.1.1 in /usr/local/lib/python3.11/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (0.1.1)\n",
760
+ "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.11/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (4.12.3)\n",
761
+ "Requirement already satisfied: bleach!=5.0.0 in /usr/local/lib/python3.11/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (6.1.0)\n",
762
+ "Requirement already satisfied: defusedxml in /usr/local/lib/python3.11/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (0.7.1)\n",
763
+ "Requirement already satisfied: jupyterlab-pygments in /usr/local/lib/python3.11/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (0.3.0)\n",
764
+ "Requirement already satisfied: mistune<4,>=2.0.3 in /usr/local/lib/python3.11/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (3.0.2)\n",
765
+ "Requirement already satisfied: nbclient>=0.5.0 in /usr/local/lib/python3.11/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (0.9.0)\n",
766
+ "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.11/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (1.5.1)\n",
767
+ "Requirement already satisfied: tinycss2 in /usr/local/lib/python3.11/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (1.2.1)\n",
768
+ "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/site-packages (from requests>=2.31->jupyterlab-server<3,>=2.19.0->jupyterlab) (2.1.1)\n",
769
+ "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/site-packages (from requests>=2.31->jupyterlab-server<3,>=2.19.0->jupyterlab) (2.2.0)\n",
770
+ "Requirement already satisfied: argon2-cffi-bindings in /usr/local/lib/python3.11/site-packages (from argon2-cffi->jupyter-server<3,>=2.4.0->jupyterlab) (21.2.0)\n",
771
+ "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.11/site-packages (from bleach!=5.0.0->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (1.16.0)\n",
772
+ "Requirement already satisfied: webencodings in /usr/local/lib/python3.11/site-packages (from bleach!=5.0.0->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (0.5.1)\n",
773
+ "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython!=2.1.4,!=2.1.5,!=2.1.6->nbdime~=4.0.1->jupyterlab-git)\n",
774
+ " Downloading smmap-5.0.1-py3-none-any.whl.metadata (4.3 kB)\n",
775
+ "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.11/site-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel->jupyterlab) (0.8.3)\n",
776
+ "Requirement already satisfied: fqdn in /usr/local/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (1.5.1)\n",
777
+ "Requirement already satisfied: isoduration in /usr/local/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (20.11.0)\n",
778
+ "Requirement already satisfied: jsonpointer>1.13 in /usr/local/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (2.4)\n",
779
+ "Requirement already satisfied: uri-template in /usr/local/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (1.3.0)\n",
780
+ "Requirement already satisfied: webcolors>=1.11 in /usr/local/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (1.13)\n",
781
+ "Requirement already satisfied: wcwidth in /usr/local/lib/python3.11/site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython>=7.23.1->ipykernel->jupyterlab) (0.2.13)\n",
782
+ "Requirement already satisfied: cffi>=1.0.1 in /usr/local/lib/python3.11/site-packages (from argon2-cffi-bindings->argon2-cffi->jupyter-server<3,>=2.4.0->jupyterlab) (1.16.0)\n",
783
+ "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.11/site-packages (from beautifulsoup4->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->jupyterlab) (2.5)\n",
784
+ "Requirement already satisfied: executing>=1.2.0 in /usr/local/lib/python3.11/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyterlab) (2.0.1)\n",
785
+ "Requirement already satisfied: asttokens>=2.1.0 in /usr/local/lib/python3.11/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyterlab) (2.4.1)\n",
786
+ "Requirement already satisfied: pure-eval in /usr/local/lib/python3.11/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyterlab) (0.2.2)\n",
787
+ "Requirement already satisfied: pycparser in /usr/local/lib/python3.11/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->jupyter-server<3,>=2.4.0->jupyterlab) (2.21)\n",
788
+ "Requirement already satisfied: arrow>=0.15.0 in /usr/local/lib/python3.11/site-packages (from isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (1.3.0)\n",
789
+ "Requirement already satisfied: types-python-dateutil>=2.8.10 in /usr/local/lib/python3.11/site-packages (from arrow>=0.15.0->isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab) (2.8.19.20240106)\n",
790
+ "Downloading jupyterlab-4.1.1-py3-none-any.whl (11.4 MB)\n",
791
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.4/11.4 MB\u001b[0m \u001b[31m474.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
792
+ "\u001b[?25hDownloading jupyterlab_git-0.50.0-py3-none-any.whl (1.2 MB)\n",
793
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m548.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
794
+ "\u001b[?25hDownloading nbdime-4.0.1-py3-none-any.whl (5.9 MB)\n",
795
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.9/5.9 MB\u001b[0m \u001b[31m217.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
796
+ "\u001b[?25hDownloading GitPython-3.1.42-py3-none-any.whl (195 kB)\n",
797
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m195.4/195.4 kB\u001b[0m \u001b[31m402.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
798
+ "\u001b[?25hDownloading gitdb-4.0.11-py3-none-any.whl (62 kB)\n",
799
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m339.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
800
+ "\u001b[?25hDownloading smmap-5.0.1-py3-none-any.whl (24 kB)\n",
801
+ "Installing collected packages: smmap, colorama, gitdb, gitpython, jupyter-server-mathjax, nbdime, jupyterlab, jupyterlab-git\n",
802
+ " Attempting uninstall: jupyterlab\n",
803
+ " Found existing installation: jupyterlab 4.1.0\n",
804
+ " Uninstalling jupyterlab-4.1.0:\n",
805
+ " Successfully uninstalled jupyterlab-4.1.0\n",
806
+ "Successfully installed colorama-0.4.6 gitdb-4.0.11 gitpython-3.1.42 jupyter-server-mathjax-0.2.6 jupyterlab-4.1.1 jupyterlab-git-0.50.0 nbdime-4.0.1 smmap-5.0.1\n",
807
+ "Note: you may need to restart the kernel to use updated packages.\n"
808
+ ]
809
+ }
810
+ ],
811
+ "source": [
812
+ "pip install --upgrade jupyterlab jupyterlab-git"
813
+ ]
814
+ },
815
+ {
816
+ "cell_type": "code",
817
+ "execution_count": 14,
818
+ "id": "05d58512-a9e2-4319-88bf-9331c6a0584c",
819
+ "metadata": {},
820
+ "outputs": [
821
+ {
822
+ "name": "stdout",
823
+ "output_type": "stream",
824
+ "text": [
825
+ "\n",
826
+ " _| _| _| _| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _|_|_|_| _|_| _|_|_| _|_|_|_|\n",
827
+ " _| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|\n",
828
+ " _|_|_|_| _| _| _| _|_| _| _|_| _| _| _| _| _| _|_| _|_|_| _|_|_|_| _| _|_|_|\n",
829
+ " _| _| _| _| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|\n",
830
+ " _| _| _|_| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _| _| _| _|_|_| _|_|_|_|\n",
831
+ "\n",
832
+ " To login, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .\n"
833
+ ]
834
+ },
835
+ {
836
+ "name": "stdin",
837
+ "output_type": "stream",
838
+ "text": [
839
+ "Token: ········\n",
840
+ "Add token as git credential? (Y/n) n\n"
841
+ ]
842
+ },
843
+ {
844
+ "name": "stdout",
845
+ "output_type": "stream",
846
+ "text": [
847
+ "Token is valid (permission: write).\n",
848
+ "Your token has been saved to /root/.cache/huggingface/token\n",
849
+ "Login successful\n"
850
+ ]
851
+ }
852
+ ],
853
+ "source": [
854
+ "from huggingface_hub import interpreter_login\n",
855
+ "interpreter_login()"
856
+ ]
857
+ },
858
+ {
859
+ "cell_type": "code",
860
+ "execution_count": 15,
861
+ "id": "bcb01d1b-9a48-46fe-b020-51a4d61df532",
862
+ "metadata": {
863
+ "scrolled": true
864
+ },
865
+ "outputs": [
866
+ {
867
+ "name": "stderr",
868
+ "output_type": "stream",
869
+ "text": [
870
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
871
+ "To disable this warning, you can either:\n",
872
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
873
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
874
+ ]
875
+ },
876
+ {
877
+ "name": "stdout",
878
+ "output_type": "stream",
879
+ "text": [
880
+ "Collecting torch==2.1.0\n",
881
+ " Downloading torch-2.1.0-cp311-cp311-manylinux1_x86_64.whl.metadata (25 kB)\n",
882
+ "Collecting pytorch-lightning==1.9.4\n",
883
+ " Downloading pytorch_lightning-1.9.4-py3-none-any.whl.metadata (22 kB)\n",
884
+ "Collecting accelerate==0.21.0\n",
885
+ " Downloading accelerate-0.21.0-py3-none-any.whl.metadata (17 kB)\n",
886
+ "Collecting tokenizers==0.13.3\n",
887
+ " Downloading tokenizers-0.13.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
888
+ "Requirement already satisfied: transformers in /usr/local/lib/python3.11/site-packages (4.37.2)\n",
889
+ "Requirement already satisfied: filelock in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (3.13.1)\n",
890
+ "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (4.9.0)\n",
891
+ "Requirement already satisfied: sympy in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (1.12)\n",
892
+ "Requirement already satisfied: networkx in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (3.2.1)\n",
893
+ "Requirement already satisfied: jinja2 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (3.1.3)\n",
894
+ "Requirement already satisfied: fsspec in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (2023.10.0)\n",
895
+ "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (12.1.105)\n",
896
+ "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (12.1.105)\n",
897
+ "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (12.1.105)\n",
898
+ "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (8.9.2.26)\n",
899
+ "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (12.1.3.1)\n",
900
+ "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (11.0.2.54)\n",
901
+ "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (10.3.2.106)\n",
902
+ "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (11.4.5.107)\n",
903
+ "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (12.1.0.106)\n",
904
+ "Collecting nvidia-nccl-cu12==2.18.1 (from torch==2.1.0)\n",
905
+ " Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB)\n",
906
+ "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.11/site-packages (from torch==2.1.0) (12.1.105)\n",
907
+ "Collecting triton==2.1.0 (from torch==2.1.0)\n",
908
+ " Downloading triton-2.1.0-0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)\n",
909
+ "Requirement already satisfied: numpy>=1.17.2 in /usr/local/lib/python3.11/site-packages (from pytorch-lightning==1.9.4) (1.26.4)\n",
910
+ "Requirement already satisfied: tqdm>=4.57.0 in /usr/local/lib/python3.11/site-packages (from pytorch-lightning==1.9.4) (4.66.2)\n",
911
+ "Requirement already satisfied: PyYAML>=5.4 in /usr/local/lib/python3.11/site-packages (from pytorch-lightning==1.9.4) (6.0.1)\n",
912
+ "Collecting torchmetrics>=0.7.0 (from pytorch-lightning==1.9.4)\n",
913
+ " Downloading torchmetrics-1.3.1-py3-none-any.whl.metadata (19 kB)\n",
914
+ "Requirement already satisfied: packaging>=17.1 in /usr/local/lib/python3.11/site-packages (from pytorch-lightning==1.9.4) (23.2)\n",
915
+ "Collecting lightning-utilities>=0.6.0.post0 (from pytorch-lightning==1.9.4)\n",
916
+ " Downloading lightning_utilities-0.10.1-py3-none-any.whl.metadata (4.8 kB)\n",
917
+ "Requirement already satisfied: psutil in /usr/local/lib/python3.11/site-packages (from accelerate==0.21.0) (5.9.8)\n",
918
+ "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.1.0) (12.3.101)\n",
919
+ "Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /usr/local/lib/python3.11/site-packages (from transformers) (0.20.3)\n",
920
+ "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.11/site-packages (from transformers) (2023.12.25)\n",
921
+ "Requirement already satisfied: requests in /usr/local/lib/python3.11/site-packages (from transformers) (2.31.0)\n",
922
+ "INFO: pip is looking at multiple versions of transformers to determine which version is compatible with other requirements. This could take a while.\n",
923
+ "Collecting transformers\n",
924
+ " Downloading transformers-4.37.1-py3-none-any.whl.metadata (129 kB)\n",
925
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.4/129.4 kB\u001b[0m \u001b[31m391.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
926
+ "\u001b[?25h Downloading transformers-4.37.0-py3-none-any.whl.metadata (129 kB)\n",
927
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.4/129.4 kB\u001b[0m \u001b[31m500.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
928
+ "\u001b[?25h Downloading transformers-4.36.2-py3-none-any.whl.metadata (126 kB)\n",
929
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m126.8/126.8 kB\u001b[0m \u001b[31m407.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
930
+ "\u001b[?25h Downloading transformers-4.36.1-py3-none-any.whl.metadata (126 kB)\n",
931
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m126.8/126.8 kB\u001b[0m \u001b[31m421.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
932
+ "\u001b[?25h Downloading transformers-4.36.0-py3-none-any.whl.metadata (126 kB)\n",
933
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m126.8/126.8 kB\u001b[0m \u001b[31m467.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
934
+ "\u001b[?25h Downloading transformers-4.35.2-py3-none-any.whl.metadata (123 kB)\n",
935
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m123.5/123.5 kB\u001b[0m \u001b[31m445.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
936
+ "\u001b[?25h Downloading transformers-4.35.1-py3-none-any.whl.metadata (123 kB)\n",
937
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m123.1/123.1 kB\u001b[0m \u001b[31m279.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
938
+ "\u001b[?25hINFO: pip is still looking at multiple versions of transformers to determine which version is compatible with other requirements. This could take a while.\n",
939
+ " Downloading transformers-4.35.0-py3-none-any.whl.metadata (123 kB)\n",
940
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m123.1/123.1 kB\u001b[0m \u001b[31m312.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
941
+ "\u001b[?25h Downloading transformers-4.34.1-py3-none-any.whl.metadata (121 kB)\n",
942
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.5/121.5 kB\u001b[0m \u001b[31m331.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
943
+ "\u001b[?25h Downloading transformers-4.34.0-py3-none-any.whl.metadata (121 kB)\n",
944
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.5/121.5 kB\u001b[0m \u001b[31m243.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
945
+ "\u001b[?25h Downloading transformers-4.33.3-py3-none-any.whl.metadata (119 kB)\n",
946
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m119.9/119.9 kB\u001b[0m \u001b[31m411.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
947
+ "\u001b[?25hRequirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.11/site-packages (from transformers) (0.4.2)\n",
948
+ "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.11/site-packages (from fsspec[http]>2021.06.0->pytorch-lightning==1.9.4) (3.8.3)\n",
949
+ "Requirement already satisfied: setuptools in /usr/local/lib/python3.11/site-packages (from lightning-utilities>=0.6.0.post0->pytorch-lightning==1.9.4) (68.1.2)\n",
950
+ "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/site-packages (from jinja2->torch==2.1.0) (2.1.5)\n",
951
+ "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/site-packages (from requests->transformers) (2.1.1)\n",
952
+ "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/site-packages (from requests->transformers) (3.6)\n",
953
+ "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/site-packages (from requests->transformers) (2.2.0)\n",
954
+ "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/site-packages (from requests->transformers) (2024.2.2)\n",
955
+ "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.11/site-packages (from sympy->torch==2.1.0) (1.3.0)\n",
956
+ "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.4) (23.2.0)\n",
957
+ "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.4) (6.0.5)\n",
958
+ "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.4) (4.0.3)\n",
959
+ "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.4) (1.9.4)\n",
960
+ "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.4) (1.4.1)\n",
961
+ "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.4) (1.3.1)\n",
962
+ "Downloading torch-2.1.0-cp311-cp311-manylinux1_x86_64.whl (670.2 MB)\n",
963
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m670.2/670.2 MB\u001b[0m \u001b[31m135.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
964
+ "\u001b[?25hDownloading pytorch_lightning-1.9.4-py3-none-any.whl (827 kB)\n",
965
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m827.8/827.8 kB\u001b[0m \u001b[31m549.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
966
+ "\u001b[?25hDownloading accelerate-0.21.0-py3-none-any.whl (244 kB)\n",
967
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m244.2/244.2 kB\u001b[0m \u001b[31m480.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
968
+ "\u001b[?25hDownloading tokenizers-0.13.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
969
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m189.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
970
+ "\u001b[?25hDownloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB)\n",
971
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m209.8/209.8 MB\u001b[0m \u001b[31m244.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
972
+ "\u001b[?25hDownloading triton-2.1.0-0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)\n",
973
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m89.2/89.2 MB\u001b[0m \u001b[31m150.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
974
+ "\u001b[?25hDownloading transformers-4.33.3-py3-none-any.whl (7.6 MB)\n",
975
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.6/7.6 MB\u001b[0m \u001b[31m234.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
976
+ "\u001b[?25hDownloading lightning_utilities-0.10.1-py3-none-any.whl (24 kB)\n",
977
+ "Downloading torchmetrics-1.3.1-py3-none-any.whl (840 kB)\n",
978
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m840.4/840.4 kB\u001b[0m \u001b[31m545.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
979
+ "\u001b[?25hInstalling collected packages: tokenizers, triton, nvidia-nccl-cu12, lightning-utilities, transformers, torch, torchmetrics, accelerate, pytorch-lightning\n",
980
+ " Attempting uninstall: tokenizers\n",
981
+ " Found existing installation: tokenizers 0.15.2\n",
982
+ " Uninstalling tokenizers-0.15.2:\n",
983
+ " Successfully uninstalled tokenizers-0.15.2\n",
984
+ " Attempting uninstall: triton\n",
985
+ " Found existing installation: triton 2.2.0\n",
986
+ " Uninstalling triton-2.2.0:\n",
987
+ " Successfully uninstalled triton-2.2.0\n",
988
+ " Attempting uninstall: nvidia-nccl-cu12\n",
989
+ " Found existing installation: nvidia-nccl-cu12 2.19.3\n",
990
+ " Uninstalling nvidia-nccl-cu12-2.19.3:\n",
991
+ " Successfully uninstalled nvidia-nccl-cu12-2.19.3\n",
992
+ " Attempting uninstall: transformers\n",
993
+ " Found existing installation: transformers 4.37.2\n",
994
+ " Uninstalling transformers-4.37.2:\n",
995
+ " Successfully uninstalled transformers-4.37.2\n",
996
+ " Attempting uninstall: torch\n",
997
+ " Found existing installation: torch 2.2.0\n",
998
+ " Uninstalling torch-2.2.0:\n",
999
+ " Successfully uninstalled torch-2.2.0\n",
1000
+ " Attempting uninstall: accelerate\n",
1001
+ " Found existing installation: accelerate 0.27.2\n",
1002
+ " Uninstalling accelerate-0.27.2:\n",
1003
+ " Successfully uninstalled accelerate-0.27.2\n",
1004
+ "Successfully installed accelerate-0.21.0 lightning-utilities-0.10.1 nvidia-nccl-cu12-2.18.1 pytorch-lightning-1.9.4 tokenizers-0.13.3 torch-2.1.0 torchmetrics-1.3.1 transformers-4.33.3 triton-2.1.0\n"
1005
+ ]
1006
+ }
1007
+ ],
1008
+ "source": [
1009
+ " !pip install torch==2.1.0 pytorch-lightning==1.9.4 accelerate==0.21.0 tokenizers==0.13.3 transformers"
1010
+ ]
1011
+ },
1012
+ {
1013
+ "cell_type": "code",
1014
+ "execution_count": 22,
1015
+ "id": "d76b4865-df75-4793-b8b9-d97523445945",
1016
+ "metadata": {},
1017
+ "outputs": [],
1018
+ "source": [
1019
+ "model.enable_input_require_grads()"
1020
+ ]
1021
+ },
1022
+ {
1023
+ "cell_type": "code",
1024
+ "execution_count": null,
1025
+ "id": "3bf553b6-b26c-49c3-9407-74c8d53a395e",
1026
+ "metadata": {
1027
+ "scrolled": true
1028
+ },
1029
+ "outputs": [
1030
+ {
1031
+ "data": {
1032
+ "text/html": [
1033
+ "\n",
1034
+ " <div>\n",
1035
+ " \n",
1036
+ " <progress value='1991' max='2000' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
1037
+ " [1991/2000 07:21 < 00:01, 4.50 it/s, Epoch 0.49/1]\n",
1038
+ " </div>\n",
1039
+ " <table border=\"1\" class=\"dataframe\">\n",
1040
+ " <thead>\n",
1041
+ " <tr style=\"text-align: left;\">\n",
1042
+ " <th>Step</th>\n",
1043
+ " <th>Training Loss</th>\n",
1044
+ " </tr>\n",
1045
+ " </thead>\n",
1046
+ " <tbody>\n",
1047
+ " <tr>\n",
1048
+ " <td>100</td>\n",
1049
+ " <td>1.383200</td>\n",
1050
+ " </tr>\n",
1051
+ " <tr>\n",
1052
+ " <td>200</td>\n",
1053
+ " <td>1.200800</td>\n",
1054
+ " </tr>\n",
1055
+ " <tr>\n",
1056
+ " <td>300</td>\n",
1057
+ " <td>1.195300</td>\n",
1058
+ " </tr>\n",
1059
+ " <tr>\n",
1060
+ " <td>400</td>\n",
1061
+ " <td>1.124500</td>\n",
1062
+ " </tr>\n",
1063
+ " <tr>\n",
1064
+ " <td>500</td>\n",
1065
+ " <td>1.138700</td>\n",
1066
+ " </tr>\n",
1067
+ " <tr>\n",
1068
+ " <td>600</td>\n",
1069
+ " <td>1.113100</td>\n",
1070
+ " </tr>\n",
1071
+ " <tr>\n",
1072
+ " <td>700</td>\n",
1073
+ " <td>1.146400</td>\n",
1074
+ " </tr>\n",
1075
+ " <tr>\n",
1076
+ " <td>800</td>\n",
1077
+ " <td>1.128400</td>\n",
1078
+ " </tr>\n",
1079
+ " <tr>\n",
1080
+ " <td>900</td>\n",
1081
+ " <td>1.086100</td>\n",
1082
+ " </tr>\n",
1083
+ " <tr>\n",
1084
+ " <td>1000</td>\n",
1085
+ " <td>1.096400</td>\n",
1086
+ " </tr>\n",
1087
+ " <tr>\n",
1088
+ " <td>1100</td>\n",
1089
+ " <td>1.111300</td>\n",
1090
+ " </tr>\n",
1091
+ " <tr>\n",
1092
+ " <td>1200</td>\n",
1093
+ " <td>1.108000</td>\n",
1094
+ " </tr>\n",
1095
+ " <tr>\n",
1096
+ " <td>1300</td>\n",
1097
+ " <td>1.124300</td>\n",
1098
+ " </tr>\n",
1099
+ " <tr>\n",
1100
+ " <td>1400</td>\n",
1101
+ " <td>1.106300</td>\n",
1102
+ " </tr>\n",
1103
+ " <tr>\n",
1104
+ " <td>1500</td>\n",
1105
+ " <td>1.100400</td>\n",
1106
+ " </tr>\n",
1107
+ " <tr>\n",
1108
+ " <td>1600</td>\n",
1109
+ " <td>1.142600</td>\n",
1110
+ " </tr>\n",
1111
+ " <tr>\n",
1112
+ " <td>1700</td>\n",
1113
+ " <td>1.081100</td>\n",
1114
+ " </tr>\n",
1115
+ " <tr>\n",
1116
+ " <td>1800</td>\n",
1117
+ " <td>1.090200</td>\n",
1118
+ " </tr>\n",
1119
+ " <tr>\n",
1120
+ " <td>1900</td>\n",
1121
+ " <td>1.111600</td>\n",
1122
+ " </tr>\n",
1123
+ " </tbody>\n",
1124
+ "</table><p>"
1125
+ ],
1126
+ "text/plain": [
1127
+ "<IPython.core.display.HTML object>"
1128
+ ]
1129
+ },
1130
+ "metadata": {},
1131
+ "output_type": "display_data"
1132
+ }
1133
+ ],
1134
+ "source": [
1135
+ "trainer = Trainer(\n",
1136
+ " model=model,\n",
1137
+ " train_dataset=tokenized_data,\n",
1138
+ " args=training_arguments,\n",
1139
+ " data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False)\n",
1140
+ ")\n",
1141
+ "trainer.train()"
1142
+ ]
1143
+ },
1144
+ {
1145
+ "cell_type": "code",
1146
+ "execution_count": null,
1147
+ "id": "263cc15e-8e9d-4bd8-9708-ec1638bc1165",
1148
+ "metadata": {},
1149
+ "outputs": [],
1150
+ "source": [
1151
+ "from peft import PeftModel\n",
1152
+ "from transformers import AutoModelForCausalLM\n",
1153
+ "import torch\n",
1154
+ "model = AutoModelForCausalLM.from_pretrained(\"microsoft/phi-2\", trust_remote_code=True, torch_dtype=torch.float32)\n",
1155
+ "peft_model = PeftModel.from_pretrained(model, \"aissatoubalde/lab\", from_transformers=True)\n",
1156
+ "model = peft_model.merge_and_unload()\n",
1157
+ "model"
1158
+ ]
1159
+ },
1160
+ {
1161
+ "cell_type": "code",
1162
+ "execution_count": null,
1163
+ "id": "8eef4f4f-52da-4ba9-8a22-2b7874420562",
1164
+ "metadata": {
1165
+ "scrolled": true
1166
+ },
1167
+ "outputs": [],
1168
+ "source": [
1169
+ "model.push_to_hub('aissatoubalde/lab')"
1170
+ ]
1171
+ },
1172
+ {
1173
+ "cell_type": "code",
1174
+ "execution_count": null,
1175
+ "id": "fa43c8e1-945e-41ef-b2f6-e90db1f50140",
1176
+ "metadata": {},
1177
+ "outputs": [],
1178
+ "source": []
1179
+ }
1180
+ ],
1181
+ "metadata": {
1182
+ "kernelspec": {
1183
+ "display_name": "Python 3 (ipykernel)",
1184
+ "language": "python",
1185
+ "name": "python3"
1186
+ },
1187
+ "language_info": {
1188
+ "codemirror_mode": {
1189
+ "name": "ipython",
1190
+ "version": 3
1191
+ },
1192
+ "file_extension": ".py",
1193
+ "mimetype": "text/x-python",
1194
+ "name": "python",
1195
+ "nbconvert_exporter": "python",
1196
+ "pygments_lexer": "ipython3",
1197
+ "version": "3.11.5"
1198
+ }
1199
+ },
1200
+ "nbformat": 4,
1201
+ "nbformat_minor": 5
1202
+ }
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:4b6c071b2811955d6107874169c939219fa3eadea899cc5d22b1dfc67183f694
3
  size 4664
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0163af3985df48ecfa67fa29c3389ea44bde4eb9309610e6d4ef07ac16ddb9b9
3
  size 4664