Patent ID: 12205035

Although the following Detailed Description will proceed with reference being made to illustrative embodiments, many alternatives, modifications and variations thereof will be apparent to those skilled in the art.

DETAILED DESCRIPTION

The systems and methods described herein provide systems and methods for enhancing the speed and accuracy of deep learning on neural networks using a neural network training tensor that includes 16-bit floating point (FP16) numbers that share a common exponent. One potential variation includes a neural network training tensor in which each floating point number is represented in a non-IEEE standard format that includes a variable length fraction portion and a variable length exponent portion. Another potential variation includes a neural network training tensor in which each floating point number includes a single ON/OFF bit used to either (a) combine the exponent of the respective floating-point number with the shared exponent; or (b) do not combine the respective floating point exponent with the shared exponent (i.e., use only the floating point exponent). Thus the systems and methods described herein provide the advantage of FP16 computation while beneficially increasing the range of possible test values through the use of a five-bit shared exponent.

Some of the systems and methods described herein generate a neural network training tensor that includes a 6-bit shared exponent that is shared by each of a plurality of IEEE 754 compliant FP16 format numbers, each of which includes a single sign bit; a five-bit exponent; and, a 10-bit mantissa. Some of the systems and methods described herein generate a neural network training tensor that includes an exponent shared by each of a plurality of IEEE 754 non-compliant FP16 format numbers, each of which includes a single sign bit; an 8-bit exponent, and a 7-bit mantissa. Some of the systems and methods described herein generate a neural network training tensor including a plurality of FP16 numbers and a 6-bit shared exponent that is shared by all of the FP16 numbers included in the tensor. Each of the FP16 numbers may have a single sign bit, a single shared exponent ON/OFF switch, an 8-bit exponent, and a 6-bit mantissa. Some of the systems and methods described herein allow processor circuitry to adjust the value of the shared exponent to avoid potential overflow or underflow conditions. Some of the systems and methods described herein allow the processor circuitry to additionally or alternatively selectively adjust the value of the exponent and mantissa of some or all of the FP16 numbers included in the tensor to avoid potential overflow or underflow conditions. Thus, to avoid a potential overflow or underflow condition, the processor circuitry may beneficially and advantageously adjust, contemporaneous with training the neural network, some or all of: the value of the shared exponent, the value of the exponent in some or all of the FP16 numbers in the tensor, and/or the value of the mantissa in some or all of the FP16 numbers in the tensor.

A system for training a neural network is provided. The system may include: processor circuitry; and a storage device coupled to the processor circuitry, the storage device including machine readable instructions. The machine-readable instructions cause the processor circuitry to: generate a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a 6-bit shared exponent common to and shared by each of the 16-bit floating point values included in the training tensor.

A neural network training method is provided. The neural network training method may include: generating, by processor circuitry, a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a 6-bit shared exponent common to each of the 16-bit floating point values included in the training tensor; and providing as an input to the neural network at least one of the plurality of 16-bit floating point values included in the training tensor.

A neural network training system is provided. The training system may include: means for generating, by processor circuitry, a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a 6-bit shared exponent common to each of the 16-bit floating point values included in the training tensor; and means for providing as an input to the neural network at least one of the plurality of 16-bit floating point values included in the training tensor.

A non-transitory, machine-readable, storage medium is provided. The non-transitory machine-readable storage medium includes machine readable instructions that, when executed by processor circuitry, cause the processor circuitry to: generate a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a 6-bit shared exponent common to each of the 16-bit floating point values included in the training tensor.

A processor-based device for training a neural network is provided. The processor-based device may include: a printed circuit board; processor circuitry coupled to the printed circuit board; an input/output interface couplable to a neural network; and a storage device coupled to the processor circuitry, the storage device including machine readable instructions that, when executed by the processor circuitry, cause the processor circuitry to: generate a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a 6-bit shared exponent common to each of the 16-bit floating point values included in the training tensor.

FIG.1provides a high level block diagram of an illustrative system100that includes a neural network110, processor circuitry130to provide a training tensor150to the neural network110, and a storage device170to store instructions executed by the processor circuitry130, in accordance with at least one embodiment described herein. The training tensor150includes a plurality of 16-bit floating point values1521-152n(collectively, “16-bit floating point values152”) and an exponent154that is common to and shared by each of at least some of the 16-bit floating point values152. In embodiments, the processor circuitry130generates or causes the generation of the training tensor150used to train (e.g., set the weighting and/or biasing of) the neural network110. In embodiments, execution of instructions stored in, on, or about the storage device170may cause the processor circuitry130to forward propagate and/or backwards propagate the training tensor150through the neural network110.

The neural network110includes a plurality of inputs1121-112n(collectively, “inputs112”) and a plurality of outputs1141-114n(collectively, “outputs114”) that care coupled by a hidden layer116that includes a plurality of neurons1161-116n(collectively, “neurons116”). Although the hidden layer116includes only a single layer of neutrons, those of ordinary skill in the relevant arts will readily appreciate the hidden layer116may include any number of layers. In embodiments, the neural network110may include a neural network capable of deep learning, a deep neural network.

The processor circuitry130may include any number and/or combination of currently available and/or future developed electrical components, semiconductor devices, and/or logic elements capable of providing the 16-bit floating point values152and 6-bit shared exponent154included in the training tensor150to the neural network110. In some implementations, the processor circuitry130may dynamically adjust the exponent154based on one or more parameters associated with the neural network110and/or the training tensor150. For example, the processor circuitry130may execute instructions that cause an increase in the shared exponent154in response to detecting a potential overflow of one or more 16-bit floating point values152included in the training tensor150. In another example, the processor circuitry130may execute instructions that cause a decrease in the shared exponent154in response to detecting a potential underflow of one or more 16-bit floating point values152included in the training tensor150.

In embodiments, the processor circuitry130may include circuitry to calculate or otherwise determine a rate of increase or a rate of decrease of one or more 16-bit floating point values152included in the training tensor150to identify those 16-bit floating point values152demonstrating behavior indicative of a potential underflow or overflow condition. In other embodiments, the processor circuitry130may include circuitry to calculate or otherwise determine an incremental increase or decrease of one or more 16-bit floating point values152included in the training tensor150to identify those 16-bit floating point values152demonstrating behavior indicative of a potential underflow or overflow condition.

In embodiments, the processor circuitry130may forward propagate some or all of the 16-bit floating point values152included in the training tensor150through the neural network110. In embodiments, the processor circuitry130may backward or reverse propagate some or all of the 16-bit floating point values152included in the training tensor150through the neural network110. In embodiments, the processor circuitry130may include one or more general purpose systems and/or devices configured to generate the 16-bit floating point values152included in the training tensor150and/or provide the 16-bit floating point values152included in the training tensor150to the neural network110. In embodiments, the processor circuitry130may receive at least a portion of the 16-bit floating point values152included in the training tensor150from one or more external sources, such as a network connected source.

The training tensor150includes a plurality of 16-bit floating point values152. The training tensor150also includes a shared exponent154that may be shared by each of at least a portion of the 16-bit floating point values152. The shared exponent154may include a sign bit to provide a signed shared exponent154. The shared exponent154may include a 3-bit value; a 4-bit value; a 5-bit value; a 6-bit value; a 7-bit value; or an 8-bit value. For example, using an IEEE 754 standard 16-bit floating point format, each 16-bit floating point value152included in the training tensor150would include: a single sign bit; a 10-bit mantissa; and an exponent that includes the sum of the 5-bit exponent included in the 16-bit floating point value152PLUS the 5-bit shared exponent154. In embodiments, the training tensor150includes a plurality of IEEE 754 compliant half-precision (16-bit floating point) values152. In embodiments, the training tensor150includes a plurality of IEEE 754 non-compliant half-precision (16-bit floating point) values152.

The storage device170may include any number and/or combination of currently available and/or future developed data storage systems and/or devices. In embodiments, the storage device170may store machine readable instruction sets executable by the processor circuitry130. In embodiments, the storage device170may hold, store, or otherwise retain in one or more data stores, data structures, and/or databases at least some of the 16-bit floating point values152that form the training tensor150.

FIG.2is an illustrative tensor200that includes a plurality of 16-bit floating point (FP16) numbers1521-152n, each of which shares a common 6-bit exponent (5:0)154and in which each of the FP16 numbers includes: a respective single sign bit (15:15)2041-204n; a respective 8-bit exponent (14:7)2061-206n; and a respective 7-bit mantissa (6:0)2081-208n, in accordance with at least one embodiment described herein. The ability to dynamically alter one or more of: the value of the shared exponent154; the mantissa value of one or more FP16 numbers; and the exponent value of one or more FP16 number included in the training tensor200beneficially permits the processor circuitry130to proactively adjust one or more FP16 values in the tensor200to avoid potential overflow or underflow conditions during the neural network training process. Using the format depicted inFIG.2, each 16-bit floating point value152may be represented as:
(−1)S*(1·M . . .0)2*2(E . . . 0)2−Exponent Bias+Signed Shared Exponent(1)

Where the exponent bias is given by:
(2)(E-M)−1  (2)

FIG.3is another illustrative tensor300that includes a plurality of 16-bit floating point (FP16) numbers1521-152nsharing a common 6-bit exponent (5:0)154and in which each of the FP16 numbers includes: a respective single sign bit (15:15)2041-204n; a respective shared exponent ON/OFF switch bit (14:14)3021-302n; a respective 8-bit exponent (14:7)2061-206n; and a respective 6-bit mantissa (5:0)2081-208n, in accordance with at least one embodiment described herein. The ability to dynamically alter one or more of: the value of the shared exponent154; the mantissa value of one or more FP16 numbers; and the exponent value of one or more FP16 number included in the training tensor300beneficially permits the processor circuitry130to proactively adjust one or more FP16 values in the tensor300to avoid potential overflow or underflow conditions during the neural network training process. In addition, the processor circuitry130may selectively designate whether the exponent206for each of the FP16 numbers152included in the training tensor300should be combined with the shared exponent—further increasing the flexibility of the training tensor300. Using the format depicted inFIG.3, with the shared exponent switch304in the “ON” logical state, a 16-bit floating point value152may be represented as:
(−1)S*(1·M . . .0)2*2(E . . .0)2-Exponent Bias+Signed Shared Exponent(3)

Where the exponent bias is given by:
(2)(E-M)−1  (4)

Using the format depicted inFIG.3, with the shared exponent switch304in the “OFF” logical state, a 16-bit floating point value152may be represented as:
(−1)S*(1·M . . .0)2*2(E . . . 0)2−Exponent Bias(5)

Where the exponent bias is given by:
(2)(E-M)−1  (6)

FIG.4is a schematic diagram of an illustrative electronic, processor-based, device400that includes processor circuitry130used to train a neural network110coupled to the processor-based device400using a training tensor150that includes a plurality of 16-bit floating point values152as described inFIGS.1,2, and3, in accordance with at least one embodiment described herein. The processor-based device400may additionally include one or more of the following: a graphical processing unit412, a wireless input/output (I/O) interface420, a wired I/O interface430, memory circuitry440, power management circuitry450, the non-transitory storage device170, and a network interface470. The following discussion provides a brief, general description of the components forming the illustrative processor-based device400. Example, non-limiting processor-based devices400may include, but are not limited to: smartphones, wearable computers, portable computing devices, handheld computing devices, desktop computing devices, blade server devices, workstations, and similar.

The processor-based device400includes processor circuitry130capable of creating or otherwise generating the 16-bit floating point values152included in the tensor150used to train the neural network110. In embodiments, the processor-based device400may additionally include graphics processor circuitry412. In embodiments, the processor-based device400includes processor circuitry130capable of executing one or more machine-readable instruction sets414, reading data and/or instruction sets414from one or more storage devices170and writing data to the one or more storage devices170. The processor circuitry130may forward propagate some or all of the 16-bit floating point values152included in the training tensor150through the neural network110. The processor circuitry130may reverse propagate some or all of the 16-bit floating point values152included in the training tensor150through the neural network110. In embodiments, the processor circuitry130may include predictive and/or detection circuitry to predict and/or detect information and/or data indicative of a potential overflow or underflow condition in one or more of the 16-bit floating point values152included in the training tensor150.

In some embodiments, the processor-based device400includes graphics processor circuitry412capable of executing machine-readable instruction sets414and generating an output signal capable of providing a display output to a system user. Those skilled in the relevant art will appreciate that the illustrated embodiments as well as other embodiments may be practiced with other processor-based device configurations, including portable electronic or handheld electronic devices, for instance smartphones, portable computers, wearable computers, consumer electronics, personal computers (“PCs”), network PCs, minicomputers, server blades, mainframe computers, and the like. The processor circuitry130may include any number of hardwired or configurable circuits, some or all of which may include programmable and/or configurable combinations of electronic components, semiconductor devices, and/or logic elements that are disposed partially or wholly in a PC, server, or other computing system capable of executing processor-readable instructions.

The processor-based device400includes a bus or similar communications link416that communicably couples and facilitates the exchange of information and/or data between various system components including the processor circuitry130, the graphics processor circuitry412, one or more wireless I/O interfaces420, one or more wired I/O interfaces430, one or more storage devices170, and/or one or more network interfaces470. The one or more network interfaces470communicate with network472. The processor-based device400may be referred to in the singular herein, but this is not intended to limit the embodiments to a single processor-based device400, since in certain embodiments, there may be more than one processor-based device400that incorporates, includes, or contains any number of communicably coupled, collocated, or remote networked circuits or devices.

The processor circuitry130may include any number, type, or combination of currently available or future developed devices capable of executing machine-readable instruction sets. The processor circuitry130may include but is not limited to any current or future developed single- or multi-core processor or microprocessor, such as: on or more systems on a chip (SOCs); central processing units (CPUs); digital signal processors (DSPs); graphics processing units (GPUs); application-specific integrated circuits (ASICs), programmable logic units, field programmable gate arrays (FPGAs), and the like. Unless described otherwise, the construction and operation of the various blocks shown inFIG.4are of conventional design. Consequently, such blocks need not be described in further detail herein, as they will be understood by those skilled in the relevant art. The bus416that interconnects at least some of the components of the processor-based device400may employ any currently available or future developed serial or parallel bus structures or architectures.

The system memory440may include read-only memory (“ROM”)442and random access memory (“RAM”)446. A portion of the ROM442may be used to store or otherwise retain a basic input/output system (“BIOS”)444. The BIOS444provides basic functionality to the processor-based device400, for example by causing the processor circuitry130to load and/or execute one or more machine-readable instruction sets414. In embodiments, at least some of the one or more machine-readable instruction sets414cause at least a portion of the processor circuitry130to provide, create, produce, transition, and/or function as a dedicated, specific, and particular machine, for example a word processing machine, a digital image acquisition machine, a media playing machine, a gaming system, a communications device, a smartphone, or similar.

The processor-based device400may include at least one wireless input/output (I/O) interface420. The at least one wireless I/O interface420may be communicably coupled to one or more physical output devices422(tactile devices, video displays, audio output devices, hardcopy output devices, etc.). The at least one wireless I/O interface420may communicably couple to one or more physical input devices424(pointing devices, touchscreens, keyboards, tactile devices, etc.). The at least one wireless I/O interface420may include any currently available or future developed wireless I/O interface. Example wireless I/O interfaces include, but are not limited to: BLUETOOTH®, near field communication (NFC), and similar.

The processor-based device400may include one or more wired input/output (I/O) interfaces430. The at least one wired I/O interface430may be communicably coupled to one or more physical output devices422(tactile devices, video displays, audio output devices, hardcopy output devices, etc.). The at least one wired I/O interface430may be communicably coupled to one or more physical input devices424(pointing devices, touchscreens, keyboards, tactile devices, etc.). The wired I/O interface430may include any currently available or future developed I/O interface. Example wired I/O interfaces include, but are not limited to: universal serial bus (USB), IEEE 1394 (“FireWire”), and similar.

The processor-based device400may include one or more communicably coupled, non-transitory, data storage devices170. The data storage devices170may include one or more hard disk drives (HDDs) and/or one or more solid-state storage devices (SSDs). The one or more data storage devices170may include any current or future developed storage appliances, network storage devices, and/or systems. Non-limiting examples of such data storage devices170may include, but are not limited to, any current or future developed non-transitory storage appliances or devices, such as one or more magnetic storage devices, one or more optical storage devices, one or more electro-resistive storage devices, one or more molecular storage devices, one or more quantum storage devices, or various combinations thereof. In some implementations, the one or more data storage devices170may include one or more removable storage devices, such as one or more flash drives, flash memories, flash storage units, or similar appliances or devices capable of communicable coupling to and decoupling from the processor-based device400.

The one or more data storage devices170may include interfaces or controllers (not shown) communicatively coupling the respective storage device or system to the bus416. The one or more data storage devices170may store, retain, or otherwise contain machine-readable instruction sets, data structures, program modules, data stores, databases, logical structures, and/or other data useful to the processor circuitry130and/or graphics processor circuitry412and/or one or more applications executed on or by the processor circuitry130and/or graphics processor circuitry412. In some instances, one or more data storage devices170may be communicably coupled to the processor circuitry130, for example via the bus416or via one or more wired communications interfaces430(e.g., Universal Serial Bus or USB); one or more wireless communications interfaces420(e.g., Bluetooth®, Near Field Communication or NFC); and/or one or more network interfaces470(IEEE 802.3 or Ethernet, IEEE 802.11, or WiFi®, etc.).

Processor-readable instruction sets414and other programs, applications, logic sets, and/or modules may be stored in whole or in part in the system memory440. Such instruction sets414may be transferred, in whole or in part, from the one or more data storage devices170. The instruction sets414may be loaded, stored, or otherwise retained in system memory440, in whole or in part, during execution by the processor circuitry130and/or graphics processor circuitry412. The processor-readable instruction sets414may include machine-readable and/or processor-readable code, instructions, or similar logic capable of causing the processor circuitry130to dynamically adjust one or more of: the value of the shared exponent154; the mantissa value of one or more FP16 numbers208; and the exponent value of one or more FP16 numbers206included in the training tensor150. The processor-readable instruction sets414may include machine-readable and/or processor-readable code, instructions, or similar logic capable of causing the processor circuitry130to selectively and individually set the shared exponent ON/OFF switch3021-302nfor each FP 16 number1521-152nto a first logical state (“ON”) in which the shared exponent154is selectively combined with the exponent2061-206nof the respective FP16 number1521-152n. The processor-readable instruction sets414may include machine-readable and/or processor-readable code, instructions, or similar logic capable of causing the processor circuitry130to selectively and individually set the shared exponent ON/OFF switch3021-302nfor each FP 16 number1521-152nto a second logical state (“OFF”) in which the shared exponent154is not combined with the exponent2061-206nof the respective FP16 number1521-152n.

The processor-based device400may include power management circuitry450that controls one or more operational aspects of the energy storage device452. In embodiments, the energy storage device452may include one or more primary (i.e., non-rechargeable) or secondary (i.e., rechargeable) batteries or similar energy storage devices. In embodiments, the energy storage device452may include one or more supercapacitors or ultracapacitors. In embodiments, the power management circuitry450may alter, adjust, or control the flow of energy from an external power source454to the energy storage device452and/or to the processor-based device400. The power source454may include, but is not limited to, a solar power system, a commercial electric grid, a portable generator, an external energy storage device, or any combination thereof.

For convenience, the processor circuitry130, the graphics processor circuitry412, the wireless I/O interface420, the wired I/O interface430, the power management circuitry450, the storage device460, and the network interface470are illustrated as communicatively coupled to each other via the bus416, thereby providing connectivity between the above-described components. In alternative embodiments, the above-described components may be communicatively coupled in a different manner than illustrated inFIG.4. For example, one or more of the above-described components may be directly coupled to other components, or may be coupled to each other, via one or more intermediary components (not shown). In another example, one or more of the above-described components may be integrated into the processor circuitry130and/or the graphics processor circuitry412. In some embodiments, all or a portion of the bus416may be omitted and the components are coupled directly to each other using suitable wired or wireless connections.

FIG.5is a high-level flow diagram of an illustrative method500of training a neural network110using a training tensor150that includes a plurality of 16-bit floating point values152having a common, shared, exponent154, in accordance with at least one embodiment described herein. The shared exponent154beneficially increases the range of possible values in the training tensor150, thereby reducing the likelihood of an overflow or underflow condition in the training tensor150. The method500commences at502.

At504, the processor circuitry130generates a training tensor150that includes a plurality of 16-bit floating point values152. The exponent of each of at least a portion of the 16-bit floating point values152may be combined with a shared exponent154associated with the training tensor150to provide 16-bit floating point values152having an expanded range over IEEE 754 compliant 16-bit floating point values.

At506, the processor circuitry130provides at least a portion of the training tensor150to the neural network110. In embodiments, the processor circuitry130forward propagates the 16-bit floating point values152to the neural network110. In embodiments, the neural network receives output from the neural network110and back-propagates at least a portion of the received 16-bit floating point values152through the neural network. The method500concludes at508.

FIG.6is a high-level flow diagram of an illustrative method600of selectively adjusting the value of one or more of the shared exponent154, one or more FP16 mantissa values2081-208n, and one or more FP16 exponent values2061-206n, responsive to detecting data indicative of a potential overflow or underflow condition, in accordance with at least one embodiment described herein. The method600commences at602.

At604, the processor circuitry130detects data indicative of a potential overflow or underflow condition in one or more of the 16-bit floating point values152included in the training tensor150. In embodiments, the processor circuitry130may include predictive circuitry and/or comparator circuitry capable of predicting a 16-bit floating point value152during future training epochs.

At606, responsive to a determination that one or more of the 16-bit floating point values152will overflow or underflow during a future training epoch, the processor circuitry130dynamically alters the value of one or more of the shared exponent154, one or more FP16 mantissa values2081-208n, and one or more FP16 exponent values2061-206nto minimize or eliminate the possibility of an overflow or underflow condition. The method600concludes at608.

FIG.7is a high-level flow diagram of an illustrative method700of selectively combining the exponent portion of a variable bit-length exponent included in a 16-bit floating point value152with a shared exponent value based on a logical state of a shared exponent switch3021-302nbit included in each of the FP16 numbers1521-152nincluded in the training tensor300, in accordance with at least one embodiment described herein. In embodiments, each of the 16-bit floating point values1521-152nmay include a respective sign bit2041-204n, a respective shared exponent ON/OFF switch bit3021-302n, a respective 6-bit mantissa2081-208n, and a respective 8-bit exponent2081-208n. The method700beneficially permits the use of multiple different exponent values for each of the FP16 numbers1521-152nincluded in the training tensor150. The method700commences at702.

At704, the processor circuitry130selectively and individually sets the shared exponent switch bit3021-302nfor each one of the FP16 numbers1521-152nincluded in the training tensor150. In embodiments, the processor circuitry130may selectively set the shared exponent switch bit304based on a determination that the exponent value2061-206nof an FP16 number1521-152nincluded in the training tensor150is insufficient to prevent a future overflow or underflow of the respective FP16 number1521-152n. By combining the exponent bits included in the 16-bit floating point value152with the shared exponent associated with the training tensor150, the processor circuitry130minimizes or eliminates the likelihood of an overflow or underflow condition. The method700concludes at706.

FIG.8is a high-level flow diagram of an illustrative method800of selectively adjusting the value of one or more of the shared exponent154, one or more FP16 mantissa values2081-208n, and one or more FP16 exponent values2061-206n, and/or selectively controlling whether individual exponent values2061-206nfor each FP 16 number1521-152nare combined with the shared exponent154responsive to detecting data indicative of a potential overflow or underflow condition, in accordance with at least one embodiment described herein. The method800commences at802.

At804, the processor circuitry130detects data indicative of a potential overflow or underflow condition in one or more of the 16-bit floating point values1521-152nincluded in the training tensor150. In embodiments, the processor circuitry130may include predictive circuitry and/or comparator circuitry capable of predicting a 16-bit floating point value152during future training epochs.

At806, responsive to a determination that one or more of the 16-bit floating point values152will overflow or underflow during a future training epoch, the processor circuitry130dynamically alters the value of one or more of the shared exponent154, one or more FP16 mantissa values2081-208n, one or more FP16 exponent values2061-206n, and/or sets the shared exponent ON/OFF switch3021-302nto a desired logical state to minimize or eliminate the possibility of an overflow or underflow condition. The method800concludes at808.

WhileFIGS.5through8illustrate various operations according to one or more embodiments, it is to be understood that not all of the operations depicted inFIGS.5through8are necessary for other embodiments. Indeed, it is fully contemplated herein that in other embodiments of the present disclosure, the operations depicted inFIGS.5through8, and/or other operations described herein, may be combined in a manner not specifically shown in any of the drawings, but still fully consistent with the present disclosure. Thus, claims directed to features and/or operations that are not exactly shown in one drawing are deemed within the scope and content of the present disclosure.

As used in this application and in the claims, a list of items joined by the term “and/or” can mean any combination of the listed items. For example, the phrase “A, B and/or C” can mean A; B; C; A and B; A and C; B and C; or A, B and C. As used in this application and in the claims, a list of items joined by the term “at least one of” can mean any combination of the listed terms. For example, the phrases “at least one of A, B or C” can mean A; B; C; A and B; A and C; B and C; or A, B and C.

As used in any embodiment herein, the terms “system” or “module” may refer to, for example, software, firmware and/or circuitry configured to perform any of the aforementioned operations. Software may be embodied as a software package, code, instructions, instruction sets and/or data recorded on non-transitory computer readable storage mediums. Firmware may be embodied as code, instructions or instruction sets and/or data that are hard-coded (e.g., nonvolatile) in memory devices. “Circuitry”, as used in any embodiment herein, may comprise, for example, singly or in any combination, hardwired circuitry, programmable circuitry such as computer processors comprising one or more individual instruction processing cores, state machine circuitry, and/or firmware that stores instructions executed by programmable circuitry or future computing paradigms including, for example, massive parallelism, analog or quantum computing, hardware embodiments of accelerators such as neural net processors and non-silicon implementations of the above. The circuitry may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, an integrated circuit (IC), system on-chip (SoC), desktop computers, laptop computers, tablet computers, servers, smartphones, etc.

Any of the operations described herein may be implemented in a system that includes one or more mediums (e.g., non-transitory storage mediums) having stored therein, individually or in combination, instructions that when executed by one or more processors perform the methods. Here, the processor may include, for example, a server CPU, a mobile device CPU, and/or other programmable circuitry. Also, it is intended that operations described herein may be distributed across a plurality of physical devices, such as processing structures at more than one different physical location. The storage medium may include any type of tangible medium, for example, any type of disk including hard disks, floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic and static RAMs, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), flash memories, Solid State Disks (SSDs), embedded multimedia cards (eMMCs), secure digital input/output (SDIO) cards, magnetic or optical cards, or any type of media suitable for storing electronic instructions. Other embodiments may be implemented as software executed by a programmable control device.

Thus, the present disclosure is directed to systems and methods for training neural networks using a tensor that includes a plurality of 16-bit floating point values and a plurality of bits that define an exponent shared by some or all of the 16-bit floating point values included in the tensor. The 16-bit floating point values may include IEEE 754 format 16-bit floating point values and the tensor may include a plurality of bits defining the shared exponent. The tensor may include a shared exponent and 16-bit floating point values that include a variable bit-length mantissa and a variable bit-length exponent that may be dynamically set by processor circuitry. The tensor may include a shared exponent and 16-bit floating point values that include a variable bit-length mantissa; a variable bit-length exponent that may be dynamically set by processor circuitry; and a shared exponent switch set by the processor circuitry to selectively combine the 16-bit floating point value exponent with the shared exponent.

The following examples pertain to further embodiments. The following examples of the present disclosure may comprise subject material such as at least one device, a method, at least one machine-readable medium for storing instructions that when executed cause a machine to perform acts based on the method, means for performing acts based on the method and/or a system for forming magnetically lined through-holes in a semiconductor package substrate.

According to example 1, there is provided a system for training a neural network. The system may include: processor circuitry; and a storage device coupled to the processor circuitry, the storage device including machine readable instructions. The machine-readable instructions cause the processor circuitry to: generate a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a five-bit shared exponent common to each of the 16-bit floating point values included in the training tensor.

Example 2 may include elements of example 1 where each of the plurality of 16-bit floating point values included in the neural network training tensor includes: a ten bit mantissa provided by the first plurality of bits; a five bit exponent provided by the second plurality of bits; and a one-bit sign.

Example 3 may include elements of any of examples 1 or 2 where each of the plurality of 16-bit floating point values included in the neural network training tensor may include: a variable bit-length mantissa provided by the first plurality of bits; a variable bit-length exponent provided by the second plurality of bits; and a one-bit sign; where the instructions further cause the processor circuitry to select, based on one or more neural network parameters: a number of bits to represent the variable bit-length mantissa; and a number of bits to represent the variable bit-length exponent.

Example 4 may include elements of any of examples 1 through 3 where the one or more neural network parameters may include data representative of a trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

Example 5 may include elements of any of examples 1 through 4 where each of the plurality of 16-bit floating point values included in the neural network training tensor may include: the mantissa provided by the first plurality of bits; the exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point number with the shared exponent.

According to example 6, there is provided a neural network training method. The neural network training method may include: generating, by processor circuitry, a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a five-bit shared exponent common to each of the 16-bit floating point values included in the training tensor; and providing as an input to the neural network at least one of the plurality of 16-bit floating point values included in the training tensor.

Example 7 may include elements of example 6 where generating the neural network training tensor may include: generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes: a ten bit mantissa provided by the first plurality of bits; a five bit exponent provided by the second plurality of bits; and a one-bit sign.

Example 8 may include elements of any of examples 6 or 7 where generating the neural network training tensor may include: generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes: a variable bit-length mantissa provided by the first plurality of bits; a variable bit-length exponent provided by the second plurality of bits; and a one-bit sign.

Example 9 may include elements of any of examples 6 through 8 and the method may additionally include selecting, by the processor circuitry based on one or more neural network parameters: a first number of bits to represent the variable bit-length mantissa; and a second number of bits to represent the variable bit-length exponent.

Example 10 may include elements of any of examples 6 through 9 and the method may additionally include detecting, by the processor circuitry, a trend in one or more of the plurality of 16-bit floating point values included in the training tensor, the detected trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

Example 11 may include elements of any of examples 6 through 10 where selecting a first number of bits to represent the variable bit-length mantissa; and a second number of bits to represent the variable bit-length exponent may include: selecting, by the processor circuitry, a first number of bits included in the first plurality of bits and a second number of bits included in the second plurality of bits responsive to detecting the trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

Example 12 may include elements of any of examples 6 through 11 where generating the neural network training tensor may include: generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes: the mantissa provided by the first plurality of bits; the exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point number with the shared exponent.

According to example 13, there is provided a neural network training system. The training system may include: means for generating, by processor circuitry, a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a five-bit shared exponent common to each of the 16-bit floating point values included in the training tensor; and means for providing as an input to the neural network at least one of the plurality of 16-bit floating point values included in the training tensor.

Example 14 may include elements of example 13 where the means for generating the neural network training tensor may include: means for generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes: a ten bit mantissa provided by the first plurality of bits; a five bit exponent provided by the second plurality of bits; and a one-bit sign.

Example 15 may include elements of any of examples 13 or 14 where the means for generating the neural network training tensor may further include: means for generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes: a variable bit-length mantissa provided by the first plurality of bits; a variable bit-length exponent provided by the second plurality of bits; and a one-bit sign.

Example 16 may include elements of any of examples 13 through 15 and the system may further include: means for selecting: a first number of bits to represent the variable bit-length mantissa; and a second number of bits to represent the variable bit-length exponent.

Example 17 may include elements of any of examples 13 through 16 and the system may further include: means for detecting a trend in one or more of the plurality of 16-bit floating point values included in the training tensor, the detected trend indicative of at least one of: a potential underflow condition or a potential overflow condition in one or more of the 16-bit floating point values included in the neural network training tensor.

Example 18 may include elements of any of examples 13 through 17 where the means for selecting a first number of bits to represent the variable bit-length mantissa; and a second number of bits to represent the variable bit-length exponent may further include: means for selecting a first number of bits included in the first plurality of bits and a second number of bits included in the second plurality of bits responsive to detecting the trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

Example 19 may include elements of any of examples 13 through 18 where the means for generating the neural network training tensor may further include: means for generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes: the mantissa provided by the first plurality of bits; the exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point number with the shared exponent.

According to example 20, there is provided a non-transitory, computer-readable, storage medium that includes machine readable instructions that, when executed by processor circuitry, cause the processor circuitry to: generate a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a five-bit shared exponent common to each of the 16-bit floating point values included in the training tensor.

Example 21 may include elements of example 20 where the machine readable instructions that cause the processor circuitry to generate neural network training tensor that includes a plurality of 16-bit floating point values further cause the processor circuitry to: generate a neural network training tensor that includes a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a ten bit mantissa provided by the first plurality of bits; a five bit exponent provided by the second plurality of bits; and a one-bit sign.

Example 22 may include elements of examples 20 or 21 where the machine readable instructions that cause the processor circuitry to generate neural network training tensor that includes a plurality of 16-bit floating point values further cause the processor circuitry to: generate a neural network training tensor that includes a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a variable bit-length mantissa provided by the first plurality of bits; a variable bit-length exponent provided by the second plurality of bits; and a one-bit sign; and select, based on one or more neural network parameters: a number of bits to represent the variable bit-length mantissa; and a number of bits to represent the variable bit-length exponent.

Example 23 may include elements of any of examples 20 through 22 where the machine readable instructions that cause the processor circuitry to select a number of bits to represent the variable bit-length mantissa and the variable bit-length exponent based on one or more neural network parameters further cause the processor circuitry to: select a number of bits to represent the variable bit-length mantissa and the variable bit-length exponent based on one or more of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

Example 24 may include elements of any of examples 20 through 23 where the machine readable instructions that cause the processor circuitry to generate neural network training tensor that includes a plurality of 16-bit floating point values further cause the processor circuitry to: generate a neural network training tensor that includes a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: the mantissa provided by the first plurality of bits; the exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point number with the shared exponent.

According to example 25, there is provided a processor-based device. The processor-based device may include: a printed circuit board; processor circuitry coupled to the printed circuit board; an input/output interface couplable to a neural network; and a storage device coupled to the processor circuitry, the storage device including machine readable instructions that, when executed by the processor circuitry, cause the processor circuitry to: generate a neural network training tensor that includes: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a first plurality of bits that form a mantissa of the respective floating point number; and a second plurality of bits that form an exponent of the respective floating point number; and a five-bit shared exponent common to each of the 16-bit floating point values included in the training tensor.

Example 26 may include elements of example 25 where each of the plurality of 16-bit floating point values included in the neural network training tensor comprises: a ten bit mantissa provided by the first plurality of bits; a five bit exponent provided by the second plurality of bits; and a one-bit sign.

Example 27 may include elements of any of examples 25 or 26 where each of the plurality of 16-bit floating point values included in the neural network training tensor may include: a variable bit-length mantissa provided by the first plurality of bits; a variable bit-length exponent provided by the second plurality of bits; and a one-bit sign; wherein the instructions further cause the processor circuitry to select, based on one or more neural network parameters: a number of bits to represent the variable bit-length mantissa; and a number of bits to represent the variable bit-length exponent.

Example 28 may include elements of any of examples 25 through 27 where the one or more neural network parameters comprise a trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

Example 29 may include elements of any of examples 25 through 28 where each of the plurality of 16-bit floating point values included in the neural network training tensor comprises: the mantissa provided by the first plurality of bits; the exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point number with the shared exponent.

According to example 30, there is provided a tensor data structure. The tensor data structure may include: a plurality of 16-bit floating point registers, each of the plurality of 16-bit floating point registers including: a first plurality of bits corresponding to a mantissa value; a second plurality of bits corresponding to an exponent value; and a sign bit; and a shared exponent register, the shared exponent register associated with each respective one of the plurality of 16-bit floating point registers.

Example 31 may include elements of example 30 where each of the plurality of 16-bit floating point registers includes: a ten bit mantissa provided by the first plurality of bits; a five bit exponent provided by the second plurality of bits; and a one-bit sign.

Example 32 may include elements of any of examples 30 or 31 where each of the plurality of 16-bit floating point registers includes: a variable bit-length mantissa provided by the first plurality of bits; a variable bit-length exponent provided by the second plurality of bits; and a one-bit sign.

Example 33 may include elements of any of examples 30 through 32 where each of the plurality of 16-bit floating point registers includes: the mantissa provided by the first plurality of bits; the exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point number with the shared exponent.

The terms and expressions which have been employed herein are used as tennis of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described (or portions thereof), and it is recognized that various modifications are possible within the scope of the claims. Accordingly, the claims are intended to cover all such equivalents. Various features, aspects, and embodiments have been described herein. The features, aspects, and embodiments are susceptible to combination with one another as well as to variation and modification, as will be understood by those having skill in the art. The present disclosure should, therefore, be considered to encompass such combinations, variations, and modifications.

As described herein, various embodiments may be implemented using hardware elements, software elements, or any combination thereof. Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth.

Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.