Package: ds2sim
# DS2SimA simple Space Sim to experiment with machine learning.It wraps [Horde3D](http://www.horde3d.org/) with Cython and uses it to renderscenes like these.<img src="docs/img/example_scene.jpg" width="400">You can use it to create your own training data, but it is easierto use the data set from the [DS2Data](https://github.com/olitheolix/ds2data)project. Not only does that project have a representative training set of therelevant objects, it also comes with a pre-rendered flight path to test yourmodel.Once you have your model you can plug it into the viewer application (see[View The Space Simulation](#View-The-Space-Simulation) section for details),and fly through the scene while it identifies the objects. You may also replacethe manual controls with another AI to make it fly autonomously.## InstallationFirst, this will only work with an NVidia GPU, because the project uses[headless rendering](https://devblogs.nvidia.com/parallelforall/egl-eye-opengl-visualization-without-x-server/).If you do not have an NVidia card, you may still train and test ML models withthe data from the [DS2Data](https://github.com/olitheolix/ds2data) set.Next, since this is a Cython wrapper for Horde3D, you will need - you guessedit - Cython and Horde3D. To compile and install Horde3D, activate the virtualenvironment of your choice, install Cython, and then clone/compile/installHorde3D like so:```bashgit clone https://github.com/olitheolix/Horde3Dmkdir -p Horde3D/buildcd Horde3D/buildgit checkout ds2simcmake .. -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=`python -c 'import sys; print(sys.prefix)'`make install```Afterwards, install *DS2Sim* via PIP:```bashpip install ds2sim```## View the Space SimulationThis consists of two parts: a web server to supply the rendered images, and aQt application to display them.Start the web server and load the default scene:```bashds2server --default-scene```Then put the following code into a file. Note that almost the entire file isboilerplate for Qt.```pythonimport sysimport ds2sim.viewerimport numpy as npimport PyQt5.QtGui as QtGuiimport PyQt5.QtCore as QtCoreimport PyQt5.QtWidgets as QtWidgets# For convenience.QPen, QColor, QRectF = QtGui.QPen, QtGui.QColor, QtCore.QRectFDS2Text = ds2sim.viewer.DS2Textclass MyClassifier(ds2sim.viewer.ClassifierCamera):def classifyImage(self, img):pass# Qt boilerplate to start the application.app = QtWidgets.QApplication(sys.argv)widget = MyClassifier('Camera', host='127.0.0.1', port=9095)widget.show()app.exec_()```The only part here that is not boilerplate is `MyClassifier`, and even thatdoes nothing right now.When you run this program you should see the scene. Click in it, and use theESDF keys, as well as the mouse, to fly through the scene.## Plug Your ML Model Into The SimulationThe real fun is, of course, to use ML to find and identify all the cubes whileyou fly around. To do so, overload the `classifyImage` method in the previousdemo like so:```python# For convenience.QPen, QColor, QRectF = QtGui.QPen, QtGui.QColor, QtCore.QRectFDS2Text = ds2server.viewer.DS2Textclass MyClassifier(ds2sim.viewer.ClassifierCamera):def classifyImage(self, img):# `img` is always a <height, width, 3> NumPy image.assert img.dtype == np.uint8# Pass the image to your ML model.# myAwesomeClassifier(img)# Define a red bounding box.x, y, width, height = 0.3, 0.4, 0.3, 0.3bbox = [QPen(QColor(255, 0, 0)), QRectF(x, y, width, height)]# Define a green text label.x, y = 0.3, 0.4text = [QPen(QColor(100, 200, 0)), DS2Text(x, y, 'Found Something')]# Install the overlays.self.setMLOverlays([bbox, text])```The `classifyImage` method will be called for each frame. It always receivesone RGB image as a NumPy array. Pass that image to your classifier to identifythe objects in the scene.When you have found out which objects are where, you can add overlays tohighlight them. Every overlay is a 2-tuple: a `QPen` to define the colour, anda primitive to draw. Currently, `QRect` and `DS2Textures` are the onlysupported primitives. Pass all overlays to `setMLOverlays` and it will show upin the scene.The example code from above will produce an output like this.Single Frame | Spaceflight:-------------------------:|:-------------------------:<img src="docs/img/viewer_box.jpg" width="400">|<img src="docs/img/animated.gif" width="400">