Package: deep-sea-treasure
DeepSeaTreasure EnvironmentInstallationIn order to get started with this environment, you can install it using the following command:python3-mpipinstalldeep_sea_treasure--upgradeDataIf you are only interested in obtaining the Pareto-front data, you can find that  on thedata/folder:2-objective Pareto-front3-objective Pareto-frontExampleAfter installing the environment, you can get started using it like this:importpygameimportnumpyasnpimporttimeimportdeep_sea_treasurefromdeep_sea_treasureimportDeepSeaTreasureV0# Make sure experiment are reproducible, so people can use the exact same versionsprint(f"Using DST{deep_sea_treasure.__version__.VERSION}({deep_sea_treasure.__version__.COMMIT_HASH})")dst:DeepSeaTreasureV0=DeepSeaTreasureV0.new(max_steps=1000,render_treasure_values=True)dst.render()stop:bool=Falsetime_reward:int=0whilenotstop:events=pygame.event.get()action=(np.asarray([0,0,0,1,0,0,0]),np.asarray([0,0,0,1,0,0,0]))foreventinevents:ifevent.type==pygame.KEYDOWN:ifevent.key==pygame.K_LEFT:action=(np.asarray([0,0,1,0,0,0,0]),np.asarray([0,0,0,1,0,0,0]))elifevent.key==pygame.K_RIGHT:action=(np.asarray([0,0,0,0,1,0,0]),np.asarray([0,0,0,1,0,0,0]))ifevent.key==pygame.K_UP:action=(np.asarray([0,0,0,1,0,0,0]),np.asarray([0,0,1,0,0,0,0]))elifevent.key==pygame.K_DOWN:action=(np.asarray([0,0,0,1,0,0,0]),np.asarray([0,0,0,0,1,0,0]))ifevent.keyin{pygame.K_ESCAPE}:stop=Trueifevent.type==pygame.QUIT:stop=True_,reward,done,debug_info=dst.step(action)time_reward+=int(reward[1])ifdone:print(f"Found treasure worth{float(reward[0]):5.2f}after{abs(time_reward)}timesteps!")time_reward=0ifnotstop:dst.render()time.sleep(0.25)ifdone:dst.reset()This will allow you to play around in the environment, and move the sub around, finding treasures!APIThis section will provide you with a detailed description of the API we offer, and how you can use it.
In general, our API matches that of OpenAI's gym package.
This library contains 1 environment (DeepSeaTreasure-v0) and 2 wrappers (FuelWrapperandVamplewWrapper).
In order to aid reproducibility, we include a package version, and source-code commit hash in the python code (deep_sea_treasure.__version__.VERSIONanddeep_sea_treasure.__version__.COMMIT_HASH).The environment we created is a modified version of the environment originally showcased in the paper byVamplew et al.In our version, the agent can move around by providing an acceleration on both the x- and y- direction.
By default, accelerations are discrete, going from -3 to +3. It is the agent's goal to find the largest possible treasure, in the shortest amount of time.
This immediately creates a tricky situation for any agent attempting to learn in this environment,
since the agent has to deal with two conflicting objectives: time and treasure. This is also reflected in the reward we return from the environment,
since this is a 2-element vector (1 element for each objective).WrappersWhen wrapping this environment, users should take care to ensure that theirdebug_dictalways contains at least anenvvalue,
identifying the wrapper that created the dict, and aninnervalue, that contains the debugging information from the wrapped environment.
This is important if you want to use the renderer to render debug information,
since the renderer is only capable of rendering debug information from the core environment itself.ExamplefromtypingimportAny,Dict,TupleclassExampleWrapper:defstep(self,action)->Tuple[Any,Any,bool,Dict[str,Any]]:obs,rew,done,debug_dict=self.env.step(action)new_debug_dict={"env":self.__class__.__name__# = "ExampleWrapper""inner":debug_dict}returnobs,rew,done,new_debug_dictGeneralAll environments and wrappers in this repository provide areward_spaceattribute,
similar to theobservation_spaceandaction_spaceattributes normally present in gym environments.
The main purpose behind this is to allow an external observer to determine the number of objectives.DeepSeaTreasureV0fromdeep_sea_treasureimportDeepSeaTreasureV0This is the core environment, creation is handled by thenew()static method.
This is the recommended way of creating environments, since it provides meaningful arguments for changing the various settings for the environment.
Beyond this, there is also a constructor that accepts a dictionary of settings, this was mainly done to retain compatibility with RL frameworks like RLLib.
Internally, thenew()method simply fills a dictionary, and delegates to the constructor.
The constructor uses JSON schemas along with a couple of extraasserts to verify the correctness of a given configuration.
The exact schema can be found indeep_sea_treasure.schema.json, and is also returned byDeepSeaTreasureV0.schema().
A copy of the default configuration can be obtained throughDeepSeaTreasureV0.default_config().
The table below describes the various options offered by thenew()method.OptionTypeMeaningtreasure_valuesList[List[Union[List[int], float]]]This option serves two purposes: It shapes the seabed, and determines the value of each treasure. It is an array of((x, y), treasure)tuples.acceleration_levelsList[int]This option provides a list of discrete acceleration levels. Each level should be a positive integer number. Upon creation, the numbers with inverted sign will be used for accelerating left and up, while the numbers themselves signify acceleration either right (x) or down (y).implicit_collision_constraintboolWhen this option is enabled, the reward function will be reduced to below its minimum when the submarine causes a collision. This is also reflected in thereward_space.low, the values in this array will be 1 below the minimum attainable through normal actions in the environment.max_stepsintThe environment has 2 ending conditions: 1. The submarine finds a treasure 2. The maximum number of steps is reached, this option sets the maximum allowed number of steps.max_velocityfloatThe environment limits the absolute value of the submarine's maximum velocity to this number, to prevent overflow-related physics shenanigans from occurring.render_gridboolIf this option is enabled, a grid will be shown when rendering the environment, showing the discrete spaces where the submarine can reside.render_treasure_valuesboolIf this option is enabled, the value of each treasure will be rendered on top of the treasure, making the rendering slightly clearer.themeThemeWe allow the customization of the rendering through the use of theThemeclass, in conjunction with theDeepSeaTreasureV0Renderer, this option allows the user to specify theThemeto use when rendering, the defaultThemeis returned byTheme.default().It is possible to pass thedebug_infodict that thestep()method returns back to therender()method, this will display some useful debugging information on screen.Action SpaceA tuple of 2, 1-hot encoded actions.Tuple[Discrete,Discrete]The first action is the acceleration in the X-dimensions, the second action is the acceleration in the Y-dimensions.
The number of available actions for each dimension is(2 * len(acceleration_levels)) + 1, sincelen(acceleration_levels)defaults to 3, the default number of available action per dimension is 7.
The middle action always indicates a no-op (No acceleration, existing velocity will continue to move the submarine), actions with indices lower than the middle move towards the top-left of the world, actions with indices greater than the middle move towards the bottom-right of the world.
When an action would cause a collision the sub's velocity is set to 0 in both dimensions and no action occurs.Observation SpaceBoxThe observation in this environment is a 2 x N matrix, with N equal tolen(treasure_values) + 1.
The first column in the matrix (obs[:, 0]) contains the submarine's current velocity.
The next N columns contain the submarine's position, relative to each of the treasures.Reward SpaceBoxThe reward for this environment is a 2-element vector.
The first element (reward[0]) contains the treasure reward. This will always be 0, unless the submarine is on a treasure square.
The second element (reward[1]) contains the time reward, this reward is always -1, unless the submarine is on a treasures square.FuelWrapperfromdeep_sea_treasureimportFuelWrapperTheFuelWrapperis a wrapper for the DeepSeaTreasureV0 environment that adds a third objective, fuel consumption.
Fuel consumption is similar to time, in the sense that the reward for this will be negative at each timestep.
It differs from time in the sense that the consumed fuel depends on the action taken.
Accelerating 1 in either dimension will usually cost 1 fuel, but accelerating 3 usually costs 3 fuel.
The addition of this mechanic makes high accelerations followed by coasting an attractive strategy.
Creation of this wrapper is handled by thenew()static method.
This is the recommended way of creating wrappers, since it provides meaningful arguments for changing the various settings for the environment.
Beyond this, there is also a constructor that accepts agym.Envand a dictionary of settings,
this was mainly done to retain compatibility with RL frameworks like RLLib.
Internally, thenew()method simply fills a dictionary, and delegates to the constructor.
The constructor uses JSON schemas along with a couple of extraasserts to verify the correctness of a given configuration.
The exact schema can be found infuel_wrapper.schema.json.OptionTypeMeaningfuel_costList[int]The cost of eachacceleration_levelin fuel units. The no-op action is always assumed to consume no fuel. The length of this list should always match the length of theacceleration_levelslist in the core environment.Action SpaceSame as DeepSeaTreasureV0.Observation SpaceSame as DeepSeaTreasureV0.Reward SpaceBoxThe reward for this environment is a 3-element vector. The first 2 elements are identical to those in theDeepSeaTreasureV0environment.
The first element (reward[0]) contains the treasure reward. This will always be 0, unless the submarine is on a treasure square.
The second element (reward[1]) contains the time reward, this reward is always -1, unless the submarine is on a treasures square.
The third element (reward[2]) contains the fuel reward, this reward reflects the fuel consumed by the last action the agent took.VamplewWrapperfromdeep_sea_treasureimportVamplewWrapperTheVamplewWrapperis a wrapper intended to undo the modifications we made to the coreDeepSeaTreasureV0environment.
It wraps both the action and observation space, to make sure the environment matches the original setup byVamplew et al.exactly.
This means that theVamplewWrapperhas a different action and observation space from the originalDeepSeaTreasureV0environment.
TheVamplewWrappercan wrap theFuelWrapper, but not the other way around, due to action-space incompatibility.
Creation of this wrapper is handled by thenew()static method.
This is the recommended way of creating wrappers, since it provides meaningful arguments for changing the various settings for the environment.
Beyond this, there is also a constructor that accepts agym.Envand a dictionary of settings,
this was mainly done to retain compatibility with RL frameworks like RLLib.
Internally, thenew()method simply fills a dictionary, and delegates to the constructor.
The constructor uses JSON schemas along with a couple of extraasserts to verify the correctness of a given configuration.
The exact schema can be found invamplew_wrapper.schema.json.OptionTypeMeaningenable_idleboolWhen true, this option enables a 5th action in this environment, idle. This allows the submarine to sit still and do nothing.Action SpaceThe action space for the Vamplew wrapper consists of a single 1-hot encoded action.
There are 4 or 5 possible actions to take, depending on how the wrapper was configured:UpRightDownLeft(Idle)Actions are specified in this order.
Each action will cancel out all velocity from the previous action, and make the velocity in the desired direction 1.Observation SpaceBoxThe observation in this environment is a 2-element vector, containing the submarine's current row and column.Reward SpaceSame as DeepSeaTreasureV0.