Package: rl-toolkit
RL ToolkitPapersSoft Actor-CriticGeneralized State-Dependent ExplorationReverb: A framework for experience replayControlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile CriticsAcme: A Research Framework for Distributed Reinforcement LearningInstallation with PyPIOn PC AMD64 with Ubuntu/DebianInstall dependencesaptupdate-y
aptinstallswig-yInstall RL-Toolkitpip3installrl-toolkit[all]Run (forServer)python3-mrl_toolkit-c./rl_toolkit/config.yaml-eMinitaurBulletEnv-v0serverRun (forAgent)python3-mrl_toolkit-c./rl_toolkit/config.yaml-eMinitaurBulletEnv-v0agent--db_serverlocalhostRun (forLearner)python3-mrl_toolkit-c./rl_toolkit/config.yaml-eMinitaurBulletEnv-v0learner--db_server192.168.1.2Run (forTester)python3-mrl_toolkit-c./rl_toolkit/config.yaml-eMinitaurBulletEnv-v0tester-fsave/model/actor.h5On NVIDIA JetsonInstall dependencesTensorflow for JetPack, follow instructionsherefor installation.aptupdate-y
aptinstallswig-y

pip3install'tensorflow-probability==0.14.1'Install ReverbDownload Bazel 3.7.2 for arm64GitHubheremv~/Downloads/bazel-3.7.2-linux-arm64~/bin/bazel
chmod+x~/bin/bazelexportPATH=$PATH:~/binClone Reverb with version that corespond with TF verion installed on NVIDIA Jetson !gitclonehttps://github.com/deepmind/reverbcdreverb/
gitcheckoutr0.5.0# for TF 2.6.0Make changes in Reverb before building !In .bazelrc- build:manylinux2010 --crosstool_top=//third_party/toolchains/preconfig/ubuntu16.04/gcc7_manylinux2010:toolchain
+ # build:manylinux2010 --crosstool_top=//third_party/toolchains/preconfig/ubuntu16.04/gcc7_manylinux2010:toolchain

- build --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0"
+ build --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=1"

- build --copt=-mavx --copt=-DEIGEN_MAX_ALIGN_BYTES=64
+ build --copt=-DEIGEN_MAX_ALIGN_BYTES=64In WORKSPACE- PROTOC_SHA256 = "15e395b648a1a6dda8fd66868824a396e9d3e89bc2c8648e3b9ab9801bea5d55"
+ # PROTOC_SHA256 = "15e395b648a1a6dda8fd66868824a396e9d3e89bc2c8648e3b9ab9801bea5d55"
+ PROTOC_SHA256 = "7877fee5793c3aafd704e290230de9348d24e8612036f1d784c8863bc790082e"In oss_build.sh-  if [ "$python_version" = "3.7" ]; then
+  if [ "$python_version" = "3.6" ]; then
+    export PYTHON_BIN_PATH=/usr/bin/python3.6 && export PYTHON_LIB_PATH=/usr/local/lib/python3.6/dist-packages
+    ABI=cp36
+  elif [ "$python_version" = "3.7" ]; then

-  bazel test -c opt --copt=-mavx --config=manylinux2010 --test_output=errors //reverb/cc/...
+  bazel test -c opt --copt="-march=armv8-a+crypto" --test_output=errors //reverb/cc/...

# Builds Reverb and creates the wheel package.
-  bazel build -c opt --copt=-mavx $EXTRA_OPT --config=manylinux2010 reverb/pip_package:build_pip_package
+  bazel build -c opt --copt="-march=armv8-a+crypto" $EXTRA_OPT reverb/pip_package:build_pip_package
./bazel-bin/reverb/pip_package/build_pip_package --dst $OUTPUT_DIR $PIP_PKG_EXTRA_ARGSIn reverb/cc/platform/default/repo.bzlurls = [
   -        "https://github.com/protocolbuffers/protobuf/releases/download/v%s/protoc-%s-linux-x86_64.zip" % (version, version),
   +        "https://github.com/protocolbuffers/protobuf/releases/download/v%s/protoc-%s-linux-aarch_64.zip" % (version, version),
]In reverb/pip_package/build_pip_package.sh-"${PYTHON_BIN_PATH}"setup.pybdist_wheel${PKG_NAME_FLAG}${RELEASE_FLAG}${TF_VERSION_FLAG}--platmanylinux2010_x86_64>/dev/null
+"${PYTHON_BIN_PATH}"setup.pybdist_wheel${PKG_NAME_FLAG}${RELEASE_FLAG}${TF_VERSION_FLAG}>/dev/nullBuild and installbashoss_build.sh--cleantrue--tf_dep_override"tensorflow=2.6.0"--release--python"3.6"bash./bazel-bin/reverb/pip_package/build_pip_package--dst/tmp/reverb/dist/--release
pip3install/tmp/reverb/dist/dm_reverb-*Cleaningcd../
rm-Rreverb/Install RL-Toolkitpip3installrl-toolkitRun (forServer)python3-mrl_toolkit-c./rl_toolkit/config.yaml-eMinitaurBulletEnv-v0serverRun (forAgent)python3-mrl_toolkit-c./rl_toolkit/config.yaml-eMinitaurBulletEnv-v0agent--db_serverlocalhostRun (forLearner)python3-mrl_toolkit-c./rl_toolkit/config.yaml-eMinitaurBulletEnv-v0learner--db_server192.168.1.2Run (forTester)python3-mrl_toolkit-c./rl_toolkit/config.yaml-eMinitaurBulletEnv-v0tester-fsave/model/actor.h5EnvironmentsEnvironmentObservation spaceObservation boundsAction spaceAction boundsBipedalWalkerHardcore-v3(24, )[-inf, inf](4, )[-1.0, 1.0]Walker2DBulletEnv-v0(22, )[-inf, inf](6, )[-1.0, 1.0]AntBulletEnv-v0(28, )[-inf, inf](8, )[-1.0, 1.0]HalfCheetahBulletEnv-v0(26, )[-inf, inf](6, )[-1.0, 1.0]HopperBulletEnv-v0(15, )[-inf, inf](3, )[-1.0, 1.0]HumanoidBulletEnv-v0(44, )[-inf, inf](17, )[-1.0, 1.0]MinitaurBulletEnv-v0(28, )[-167.72488, 167.72488](8, )[-1.0, 1.0]ResultsEnvironmentSAC+ gSDESAC+ gSDE+ Huber lossSAC+ TQC+ gSDESAC+ TQC+ gSDE+ LogCosh+ ReverbBipedalWalkerHardcore-v313 ± 18(2)-228 ± 18(2)-Walker2DBulletEnv-v02270 ± 28(1)2732 ± 962535 ± 94(2)-AntBulletEnv-v03106 ± 61(1)3460 ± 1193700 ± 37(2)-HalfCheetahBulletEnv-v02945 ± 95(1)3003 ± 2263041 ± 157(2)-HopperBulletEnv-v02515 ± 50(1)2555 ± 4052401 ± 62(2)-HumanoidBulletEnv-v0----MinitaurBulletEnv-v0----ReleasesSAC + gSDE + Huber lossis stored here,branch r2.0SAC + TQC + gSDE + LogCosh + Reverbis stored here,branch r4.0Frameworks:Tensorflow, Reverb, OpenAI Gym, PyBullet, WanDB, OpenCVChangesv4.1.1 (September 2, 2022)update defaultconfig.yamlv4.1.0 (February 9, 2022)Features 🔊.fit()AgentCallbackv4.0.0 (February 5, 2022)Features 🔊Render environments to WanDBGrouping of runs in WanDBSampleToInsertRatio rate limiterGlobal Gradient Clipping to avoid exploding gradientsSoftplus for numerical stabilityYAML configuration fileLogCosh instead of Huber lossCritic network with Add layer applied on state & action branchesCustom uniform initializerXLA (Accelerated Linear Algebra) compilerOptimized Replay Buffer (https://github.com/deepmind/reverb/issues/90)split intoAgent,Learner,TesterandServerBug fixes 🛠️Fixed creating of saving path for modelsFixed model'ssummary()v3.2.4 (July 7, 2021)Features 🔊Reverbsetup.py(package is available on PyPI)split intoAgent,LearnerandTesterUse custom model and layer for defining Actor-CriticMultiCritic - concatenating multiple critic networks into one networkTruncated Quantile Criticsv2.0.2 (May 23, 2021)Features 🔊update DockerfileupdateREADME.mdformatted code by Black & Flake8v2.0.1 (April 27, 2021)Bug fixes 🛠️fixed Critic modelv2.0.0 (April 22, 2021)Features 🔊Add Huber lossIn test mode, rendering to the video fileNormalized observation by Min-max methodRemove TD3 algorithm