zhangpaipai commited on
Commit
72c17a0
1 Parent(s): 54e54cd

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +290 -0
README.md ADDED
@@ -0,0 +1,290 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: apache-2.0
4
+ library_name: pytorch
5
+ tags:
6
+ - deep-reinforcement-learning
7
+ - reinforcement-learning
8
+ - DI-engine
9
+ - Pendulum-v1
10
+ benchmark_name: OpenAI/Gym/ClassicControl
11
+ task_name: Pendulum-v1
12
+ pipeline_tag: reinforcement-learning
13
+ model-index:
14
+ - name: TD3
15
+ results:
16
+ - task:
17
+ type: reinforcement-learning
18
+ name: reinforcement-learning
19
+ dataset:
20
+ name: OpenAI/Gym/ClassicControl-Pendulum-v1
21
+ type: OpenAI/Gym/ClassicControl-Pendulum-v1
22
+ metrics:
23
+ - type: mean_reward
24
+ value: -200.85 +/- 155.78
25
+ name: mean_reward
26
+ ---
27
+
28
+ # Play **Pendulum-v1** with **TD3** Policy
29
+
30
+ ## Model Description
31
+ <!-- Provide a longer summary of what this model is. -->
32
+ This is a simple **TD3** implementation to OpenAI/Gym/ClassicControl **Pendulum-v1** using the [DI-engine library](https://github.com/opendilab/di-engine) and the [DI-zoo](https://github.com/opendilab/DI-engine/tree/main/dizoo).
33
+
34
+ **DI-engine** is a python library for solving general decision intelligence problems, which is based on implementations of reinforcement learning framework using PyTorch or JAX. This library aims to standardize the reinforcement learning framework across different algorithms, benchmarks, environments, and to support both academic researches and prototype applications. Besides, self-customized training pipelines and applications are supported by reusing different abstraction levels of DI-engine reinforcement learning framework.
35
+
36
+
37
+
38
+ ## Model Usage
39
+ ### Install the Dependencies
40
+ <details close>
41
+ <summary>(Click for Details)</summary>
42
+
43
+ ```shell
44
+ # install huggingface_ding
45
+ git clone https://github.com/opendilab/huggingface_ding.git
46
+ pip3 install -e ./huggingface_ding/
47
+ # install environment dependencies if needed
48
+ pip3 install DI-engine[common_env]
49
+ ```
50
+ </details>
51
+
52
+ ### Git Clone from Huggingface and Run the Model
53
+
54
+ <details close>
55
+ <summary>(Click for Details)</summary>
56
+
57
+ ```shell
58
+ # running with trained model
59
+ python3 -u run.py
60
+ ```
61
+ **run.py**
62
+ ```python
63
+ from ding.bonus import TD3Agent
64
+ from ding.config import Config
65
+ from easydict import EasyDict
66
+ import torch
67
+
68
+ # Pull model from files which are git cloned from huggingface
69
+ policy_state_dict = torch.load("pytorch_model.bin", map_location=torch.device("cpu"))
70
+ cfg = EasyDict(Config.file_to_dict("policy_config.py"))
71
+ # Instantiate the agent
72
+ agent = TD3Agent(
73
+ env="pendulum", exp_name="Pendulum-v1-TD3", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
74
+ )
75
+ # Continue training
76
+ agent.train(step=5000)
77
+ # Render the new agent performance
78
+ agent.deploy(enable_save_replay=True)
79
+
80
+ ```
81
+ </details>
82
+
83
+ ### Run Model by Using Huggingface_ding
84
+
85
+ <details close>
86
+ <summary>(Click for Details)</summary>
87
+
88
+ ```shell
89
+ # running with trained model
90
+ python3 -u run.py
91
+ ```
92
+ **run.py**
93
+ ```python
94
+ from ding.bonus import TD3Agent
95
+ from huggingface_ding import pull_model_from_hub
96
+
97
+ # Pull model from Hugggingface hub
98
+ policy_state_dict, cfg = pull_model_from_hub(repo_id="OpenDILabCommunity/Pendulum-v1-TD3")
99
+ # Instantiate the agent
100
+ agent = TD3Agent(
101
+ env="pendulum",
102
+ exp_name="Pendulum-v1-TD3",
103
+ cfg=cfg.exp_config,
104
+ policy_state_dict=policy_state_dict
105
+ )
106
+ # Continue training
107
+ agent.train(step=5000)
108
+ # Render the new agent performance
109
+ agent.deploy(enable_save_replay=True)
110
+
111
+ ```
112
+ </details>
113
+
114
+ ## Model Training
115
+
116
+ ### Train the Model and Push to Huggingface_hub
117
+
118
+ <details close>
119
+ <summary>(Click for Details)</summary>
120
+
121
+ ```shell
122
+ #Training Your Own Agent
123
+ python3 -u train.py
124
+ ```
125
+ **train.py**
126
+ ```python
127
+ from ding.bonus import TD3Agent
128
+ from huggingface_ding import push_model_to_hub
129
+
130
+ # Instantiate the agent
131
+ agent = TD3Agent("pendulum", exp_name="Pendulum-v1-TD3")
132
+ # Train the agent
133
+ return_ = agent.train(step=int(4000000))
134
+ # Push model to huggingface hub
135
+ push_model_to_hub(
136
+ agent=agent.best,
137
+ env_name="OpenAI/Gym/ClassicControl",
138
+ task_name="Pendulum-v1",
139
+ algo_name="TD3",
140
+ wandb_url=return_.wandb_url,
141
+ github_repo_url="https://github.com/opendilab/DI-engine",
142
+ github_doc_model_url="https://di-engine-docs.readthedocs.io/en/latest/12_policies/td3.html",
143
+ github_doc_env_url="https://di-engine-docs.readthedocs.io/en/latest/13_envs/pendulum.html",
144
+ installation_guide="pip3 install DI-engine[common_env]",
145
+ usage_file_by_git_clone="./td3/pendulum_td3_deploy.py",
146
+ usage_file_by_huggingface_ding="./td3/pendulum_td3_download.py",
147
+ train_file="./td3/pendulum_td3.py",
148
+ repo_id="OpenDILabCommunity/Pendulum-v1-TD3"
149
+ )
150
+
151
+ ```
152
+ </details>
153
+
154
+ **Configuration**
155
+ <details close>
156
+ <summary>(Click for Details)</summary>
157
+
158
+
159
+ ```python
160
+ exp_config = {
161
+ 'env': {
162
+ 'manager': {
163
+ 'episode_num': float("inf"),
164
+ 'max_retry': 1,
165
+ 'retry_type': 'reset',
166
+ 'auto_reset': True,
167
+ 'step_timeout': None,
168
+ 'reset_timeout': None,
169
+ 'retry_waiting_time': 0.1,
170
+ 'cfg_type': 'BaseEnvManagerDict'
171
+ },
172
+ 'stop_value': -250,
173
+ 'collector_env_num': 8,
174
+ 'evaluator_env_num': 5,
175
+ 'act_scale': True,
176
+ 'n_evaluator_episode': 5
177
+ },
178
+ 'policy': {
179
+ 'model': {
180
+ 'twin_critic': True,
181
+ 'obs_shape': 3,
182
+ 'action_shape': 1,
183
+ 'action_space': 'regression'
184
+ },
185
+ 'learn': {
186
+ 'learner': {
187
+ 'train_iterations': 1000000000,
188
+ 'dataloader': {
189
+ 'num_workers': 0
190
+ },
191
+ 'log_policy': True,
192
+ 'hook': {
193
+ 'load_ckpt_before_run': '',
194
+ 'log_show_after_iter': 100,
195
+ 'save_ckpt_after_iter': 10000,
196
+ 'save_ckpt_after_run': True
197
+ },
198
+ 'cfg_type': 'BaseLearnerDict'
199
+ },
200
+ 'update_per_collect': 2,
201
+ 'batch_size': 128,
202
+ 'learning_rate_actor': 0.001,
203
+ 'learning_rate_critic': 0.001,
204
+ 'ignore_done': True,
205
+ 'target_theta': 0.005,
206
+ 'discount_factor': 0.99,
207
+ 'actor_update_freq': 2,
208
+ 'noise': True,
209
+ 'noise_sigma': 0.1,
210
+ 'noise_range': {
211
+ 'min': -0.5,
212
+ 'max': 0.5
213
+ }
214
+ },
215
+ 'collect': {
216
+ 'collector': {
217
+ 'collect_print_freq': 1000
218
+ },
219
+ 'unroll_len': 1,
220
+ 'noise_sigma': 0.1,
221
+ 'n_sample': 48
222
+ },
223
+ 'eval': {
224
+ 'evaluator': {
225
+ 'eval_freq': 100,
226
+ 'render': {
227
+ 'render_freq': -1,
228
+ 'mode': 'train_iter'
229
+ },
230
+ 'cfg_type': 'InteractionSerialEvaluatorDict',
231
+ 'n_episode': 5,
232
+ 'stop_value': -250
233
+ }
234
+ },
235
+ 'other': {
236
+ 'replay_buffer': {
237
+ 'replay_buffer_size': 20000
238
+ }
239
+ },
240
+ 'on_policy': False,
241
+ 'cuda': False,
242
+ 'multi_gpu': False,
243
+ 'bp_update_sync': True,
244
+ 'traj_len_inf': False,
245
+ 'type': 'td3',
246
+ 'priority': False,
247
+ 'priority_IS_weight': False,
248
+ 'random_collect_size': 800,
249
+ 'transition_with_policy_data': False,
250
+ 'action_space': 'continuous',
251
+ 'reward_batch_norm': False,
252
+ 'multi_agent': False,
253
+ 'cfg_type': 'TD3PolicyDict'
254
+ },
255
+ 'exp_name': 'Pendulum-v1-TD3',
256
+ 'seed': 0,
257
+ 'wandb_logger': {
258
+ 'gradient_logger': True,
259
+ 'video_logger': True,
260
+ 'plot_logger': True,
261
+ 'action_logger': True,
262
+ 'return_logger': False
263
+ }
264
+ }
265
+
266
+ ```
267
+ </details>
268
+
269
+ **Training Procedure**
270
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
271
+ - **Weights & Biases (wandb):** [monitor link](https://wandb.ai/zhangpaipai/Pendulum-v1-TD3)
272
+
273
+ ## Model Information
274
+ <!-- Provide the basic links for the model. -->
275
+ - **Github Repository:** [repo link](https://github.com/opendilab/DI-engine)
276
+ - **Doc**: [DI-engine-docs Algorithm link](https://di-engine-docs.readthedocs.io/en/latest/12_policies/td3.html)
277
+ - **Configuration:** [config link](https://huggingface.co/OpenDILabCommunity/Pendulum-v1-TD3/blob/main/policy_config.py)
278
+ - **Demo:** [video](https://huggingface.co/OpenDILabCommunity/Pendulum-v1-TD3/blob/main/replay.mp4)
279
+ <!-- Provide the size information for the model. -->
280
+ - **Parameters total size:** 53.01 KB
281
+ - **Last Update Date:** 2023-04-29
282
+
283
+ ## Environments
284
+ <!-- Address questions around what environment the model is intended to be trained and deployed at, including the necessary information needed to be provided for future users. -->
285
+ - **Benchmark:** OpenAI/Gym/ClassicControl
286
+ - **Task:** Pendulum-v1
287
+ - **Gym version:** 0.25.1
288
+ - **DI-engine version:** v0.4.7
289
+ - **PyTorch version:** 1.7.1
290
+ - **Doc**: [DI-engine-docs Environments link](https://di-engine-docs.readthedocs.io/en/latest/13_envs/pendulum.html)