Upload README.md with huggingface_hub

673e492 verified 5 months ago

No virus

9.96 kB

	---
	language: en
	license: apache-2.0
	library_name: pytorch
	tags:
	- deep-reinforcement-learning
	- reinforcement-learning
	- DI-engine
	- TicTacToe-play-with-bot
	benchmark_name: OpenAI/Gym/Atari
	task_name: TicTacToe-play-with-bot
	pipeline_tag: reinforcement-learning
	model-index:
	- name: SampledAlphaZero
	results:
	- task:
	type: reinforcement-learning
	name: reinforcement-learning
	dataset:
	name: TicTacToe-play-with-bot
	type: TicTacToe-play-with-bot
	metrics:
	- type: mean_reward
	value: 0.3 +/- 0.64
	name: mean_reward
	---

	# Play TicTacToe-play-with-bot with SampledAlphaZero Policy

	## Model Description
	<!-- Provide a longer summary of what this model is. -->

	This implementation applies SampledAlphaZero to the OpenAI/Gym/Atari TicTacToe-play-with-bot environment using [LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine).

	LightZero is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).

	## Model Usage
	### Install the Dependencies
	<details close>
	<summary>(Click for Details)</summary>

	```shell
	# install huggingface_ding
	git clone https://github.com/opendilab/huggingface_ding.git
	pip3 install -e ./huggingface_ding/
	# install environment dependencies if needed

	pip3 install DI-engine[common_env,video]
	pip3 install LightZero

	```
	</details>

	### Git Clone from Huggingface and Run the Model

	<details close>
	<summary>(Click for Details)</summary>

	```shell
	# running with trained model
	python3 -u run.py
	```
	run.py
	```python
	from lzero.agent import SampledAlphaZeroAgent
	from ding.config import Config
	from easydict import EasyDict
	import torch

	# Pull model from files which are git cloned from huggingface
	policy_state_dict = torch.load("pytorch_model.bin", map_location=torch.device("cpu"))
	cfg = EasyDict(Config.file_to_dict("policy_config.py").cfg_dict)
	# Instantiate the agent
	agent = SampledAlphaZeroAgent(
	env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-SampledAlphaZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
	)
	# Continue training
	agent.train(step=5000)
	# Render the new agent performance
	agent.deploy(enable_save_replay=True)

	```
	</details>

	### Run Model by Using Huggingface_ding

	<details close>
	<summary>(Click for Details)</summary>

	```shell
	# running with trained model
	python3 -u run.py
	```
	run.py
	```python
	from lzero.agent import SampledAlphaZeroAgent
	from huggingface_ding import pull_model_from_hub

	# Pull model from Hugggingface hub
	policy_state_dict, cfg = pull_model_from_hub(repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero")
	# Instantiate the agent
	agent = SampledAlphaZeroAgent(
	env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-SampledAlphaZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
	)
	# Continue training
	agent.train(step=5000)
	# Render the new agent performance
	agent.deploy(enable_save_replay=True)

	```
	</details>

	## Model Training

	### Train the Model and Push to Huggingface_hub

	<details close>
	<summary>(Click for Details)</summary>

	```shell
	#Training Your Own Agent
	python3 -u train.py
	```
	train.py
	```python
	from lzero.agent import SampledAlphaZeroAgent
	from huggingface_ding import push_model_to_hub

	# Instantiate the agent
	agent = SampledAlphaZeroAgent(env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-SampledAlphaZero")
	# Train the agent
	return_ = agent.train(step=int(500000))
	# Push model to huggingface hub
	push_model_to_hub(
	agent=agent.best,
	env_name="OpenAI/Gym/Atari",
	task_name="TicTacToe-play-with-bot",
	algo_name="SampledAlphaZero",
	github_repo_url="https://github.com/opendilab/LightZero",
	github_doc_model_url=None,
	github_doc_env_url=None,
	installation_guide='''
	pip3 install DI-engine[common_env,video]
	pip3 install LightZero
	''',
	usage_file_by_git_clone="./sampled_alphazero/tictactoe_play_with_bot_sampled_alphazero_deploy.py",
	usage_file_by_huggingface_ding="./sampled_alphazero/tictactoe_play_with_bot_sampled_alphazero_download.py",
	train_file="./sampled_alphazero/tictactoe_play_with_bot_sampled_alphazero.py",
	repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero",
	platform_info="[LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine)",
	model_description="LightZero is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).",
	create_repo=True
	)

	```
	</details>

	Configuration
	<details close>
	<summary>(Click for Details)</summary>


	```python
	exp_config = {
	'main_config': {
	'exp_name': 'TicTacToe-play-with-bot-SampledAlphaZero',
	'seed': 0,
	'env': {
	'env_id': 'TicTacToe-play-with-bot',
	'board_size': 3,
	'battle_mode': 'play_with_bot_mode',
	'bot_action_type': 'v0',
	'channel_last': False,
	'collector_env_num': 8,
	'evaluator_env_num': 5,
	'n_evaluator_episode': 5,
	'manager': {
	'shared_memory': False
	},
	'agent_vs_human': False,
	'prob_random_agent': 0,
	'prob_expert_agent': 0,
	'scale': True,
	'alphazero_mcts_ctree': False,
	'save_replay_gif': False,
	'replay_path_gif': './replay_gif'
	},
	'policy': {
	'on_policy': False,
	'cuda': True,
	'multi_gpu': False,
	'bp_update_sync': True,
	'traj_len_inf': False,
	'model': {
	'observation_shape': [3, 3, 3],
	'action_space_size': 9,
	'num_res_blocks': 1,
	'num_channels': 16,
	'fc_value_layers': [8],
	'fc_policy_layers': [8]
	},
	'torch_compile': False,
	'tensor_float_32': False,
	'sampled_algo': False,
	'gumbel_algo': False,
	'update_per_collect': 50,
	'model_update_ratio': 0.1,
	'batch_size': 256,
	'optim_type': 'Adam',
	'learning_rate': 0.003,
	'weight_decay': 0.0001,
	'momentum': 0.9,
	'grad_clip_value': 0.5,
	'value_weight': 1.0,
	'collector_env_num': 8,
	'evaluator_env_num': 5,
	'lr_piecewise_constant_decay': False,
	'threshold_training_steps_for_final_lr': 500000,
	'manual_temperature_decay': False,
	'threshold_training_steps_for_final_temperature': 100000,
	'fixed_temperature_value': 0.25,
	'mcts': {
	'num_simulations': 25
	},
	'other': {
	'replay_buffer': {
	'replay_buffer_size': 1000000,
	'save_episode': False
	}
	},
	'cfg_type': 'AlphaZeroPolicyDict',
	'mcts_ctree': False,
	'simulation_env_name': 'tictactoe',
	'simulation_env_config_type': 'play_with_bot',
	'board_size': 3,
	'entropy_weight': 0.0,
	'n_episode': 8,
	'eval_freq': 2000
	},
	'wandb_logger': {
	'gradient_logger': False,
	'video_logger': False,
	'plot_logger': False,
	'action_logger': False,
	'return_logger': False
	}
	},
	'create_config': {
	'env': {
	'type': 'tictactoe',
	'import_names': ['zoo.board_games.tictactoe.envs.tictactoe_env']
	},
	'env_manager': {
	'type': 'subprocess'
	},
	'policy': {
	'type': 'alphazero',
	'import_names': ['lzero.policy.alphazero']
	},
	'collector': {
	'type': 'episode_alphazero',
	'import_names': ['lzero.worker.alphazero_collector']
	},
	'evaluator': {
	'type': 'alphazero',
	'import_names': ['lzero.worker.alphazero_evaluator']
	}
	}
	}

	```
	</details>

	Training Procedure
	<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
	- Weights & Biases (wandb): [monitor link](<TODO>)

	## Model Information
	<!-- Provide the basic links for the model. -->
	- Github Repository: [repo link](https://github.com/opendilab/LightZero)
	- Doc: [Algorithm link](<TODO>)
	- Configuration: [config link](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero/blob/main/policy_config.py)
	- Demo: [video](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero/blob/main/replay.mp4)
	<!-- Provide the size information for the model. -->
	- Parameters total size: 51.13 KB
	- Last Update Date: 2024-02-01

	## Environments
	<!-- Address questions around what environment the model is intended to be trained and deployed at, including the necessary information needed to be provided for future users. -->
	- Benchmark: OpenAI/Gym/Atari
	- Task: TicTacToe-play-with-bot
	- Gym version: 0.25.1
	- DI-engine version: v0.5.0
	- PyTorch version: 2.0.1+cu117
	- Doc: [Environments link](<TODO>)