Project: Apache Hadoop (App Definition and Development / Database)
File: DevelopYourOwnDevicePlugin.md

<! Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> A device plugin is loaded into the framework when starting NM. Your plugin class only needs to consider two interfaces provided by the framework. The `DevicePlugin` is a must to implement and the `DevicePluginScheduler` is optional. ``` / A must interface for vendor plugin to implement. / public interface DevicePlugin { / Called first when device plugin framework wants to register. @return DeviceRegisterRequest {@link DeviceRegisterRequest} @throws Exception / DeviceRegisterRequest getRegisterRequestInfo() throws Exception; / Called when update node resource. @return a set of {@link Device}, {@link java.util.TreeSet} recommended @throws Exception / Set<Device> getDevices() throws Exception; / Asking how these devices should be prepared/used before/when container launch. A plugin can do some tasks in its own or define it in DeviceRuntimeSpec to let the framework do it. For instance, define {@code VolumeSpec} to let the framework to create volume before running container. * @param allocatedDevices A set of allocated {@link Device}. @param yarnRuntime Indicate which runtime YARN will use Could be {@code RUNTIMEDEFAULT} or {@code RUNTIMEDOCKER} in {@link DeviceRuntimeSpec} constants. The default means YARN's non-docker container runtime is used. The docker means YARN's docker container runtime is used. @return a {@link DeviceRuntimeSpec} description about environment, {@link VolumeSpec}, {@link MountVolumeSpec}. etc @throws Exception / DeviceRuntimeSpec onDevicesAllocated(Set<Device>; allocatedDevices, YarnRuntimeType yarnRuntime) throws Exception; / Called after device released. @param releasedDevices A set of released devices @throws Exception / void onDevicesReleased(Set<Device> releasedDevices) throws Exception; } ``` The above code shows the `DevicePlugin` interface you need to implement. Lets go through the methods that a your plugin should implement. getRegisterRequestInfo(): DeviceRegisterRequest getDevices: Set&lt;Device&gt; onDevicesAllocated(Set&lt;Device&gt;, YarnRuntimeType yarnRuntime): DeviceRuntimeSpec onDeviceReleased(Set&lt;Device&gt;): void The getRegisterRequestInfo interface is used for the plugin to advertise a new resource type name and then the ResourceManager. The DeviceRegisterRequest returned by the method consists a plugin version and a resource type name like nvidia.com/gpu. The getDevices interface is used to get latest vendor device list in this NM node. The resource count pre-defined in node-resources.xml will be overridden. And its recommended that the vendor plugin manages allowed devices reported to YARN in its own configuration. YARN can only have a blacklist configuration `devices.denied-numbers` in `container-executor.cfg`. In this method, you may invoke shell command or invoke RESTful/RPC to remote service to get the devices at your

Please note that the `Device` object can describe a fake device. If the major device number, minor device number and device path is left unset, the framework won't do isolation for it. This provide feasibility for user to define a fake device without real hardware. The onDevicesAllocated interface is invoked to tell the framework how to use these devices. The NM invoke this interface to let the plugin do some preparation work like create volume before container launch and give hints on how to expose the devices to container when launch it. The `DeviceRuntimeSpec` is the structure of the hints. For instance, `DeviceRuntimeSpec` can describes the container launch requirements like environment variables, device and volume mounts, Docker runtime type.etc. The onDeviceReleased interface is used for the plugin to do some cleanup work after container finish. ``` / An optional interface to implement if custom device scheduling is needed. If this is not implemented, the device framework will do scheduling. / public interface DevicePluginScheduler { / Called when allocating devices. The framework will do all device book keeping and fail recovery. So this hook could be stateless and only do scheduling based on available devices passed in. It could be invoked multiple times by the framework. The hint in environment variables passed in could be potentially used in making better scheduling decision. For instance, GPU scheduling might support different kind of policy. The container can set it through environment variables. @param availableDevices Devices allowed to be chosen from. @param count Number of device to be allocated. @param env Environment variables of the container. @return A set of {@link Device} allocated / Set<Device> allocateDevices(Set<Device> availableDevices, int count, Map<String, String> env); } ``` The above code shows the `DevicePluginScheduler` interface that you might needed if you want to arm the plugin with a more efficient scheduler. This `allocateDevices` method is invoked by YARN each time when asking the plugin's recommendation devices for one container. This interface is optional because YARN will provide a very basic scheduler. You can refer to `NvidiaGPUPluginForRuntimeV2` plugin for a plugin customized scheduler. Its scheduler is targeting for Nvidia GPU topology aware scheduling and can get considerable performance boost for the container. When developing the plugin, you need to add below dependency property into your projects's `pom.xml`. For instance, ``` <dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-server-nodemanager</artifactId> <version>3.3.0</version> <scope>provided</scope> </dependency> </dependencies> ``` And after this, you can implement the above interfaces based on classes provided in `org.apache.hadoop.yarn.server.nodemanager.api.deviceplugin`. Please note that the plugin project is coupled with the Hadoop YARN NM version. Once you build your project and package a jar which contains your plugin class and want to give it a try in your Hadoop cluster. Firstly, put the jar file under a directory in Hadooop classpath. (recommend $HADOOPCOMMANDHOME/share/hadoop/yarn). Secondly, follow the configurations described in and restart YARN.