Patent ID: 11880762
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A computer-implemented method for selecting from among multiple Graphics Processing Unit (GPU) execution modes for a Neural Network (NN) having a size greater than a threshold physical GPU memory size, the multiple GPU execution modes comprising a normal memory mode, an Out-of-Core (OoC) execution mode, and a Unified Memory (UM) mode, the method comprising:
in a multiple execution mode selection scheme having any of the multiple GPU execution modes of the normal memory mode, the OoC execution mode, and the UM mode available for initial execution of all layers of the NN, performing an initial iteration by executing all of the layers of the NN with the UM mode from among the multiple GPU execution modes to ensure memory coverage of the NN;
for each iteration of executing all of the layers of the NN:
measuring memory usage for each layer of the NN;
changing to a UM mode by copying data from a non-unified memory to a UM responsive to an occurrence of a memory shortage when executing in the normal memory mode or OoC execution mode;
determining a total memory usage by all of the layers of the NN and a maximum layer memory usage; and
selecting from the multiple GPU execution modes a GPU execution mode for each of the layers of the NN to execute with in a next iteration, wherein the normal memory mode is selected responsive to the total memory usage by all of the layers of the NN being less than a first predetermined threshold and the OoC execution mode is selected responsive to the total memory usage by all of the layers of the NN being greater than or equal to the first predetermined threshold and the maximum layer memory usage being less than a second predetermined threshold.