Patent ID: 12254399

DETAILED DESCRIPTION

Embodiments disclosed herein provide a balanced approach between fully analog or fully digital implementations for networks-on-chip (NoCs) that execute neural networks by providing power savings due to fewer data conversions with the use of an analog router and flexibility and scalability of a digital NoC at the higher level. The disclosure may include a Gaussian distribution circuit embedded in the analog router for supporting probabilistic machine learning applications. In addition, the introduction of stochastic rounding in the analog router provides higher accuracy in reduced precision neural networks as well as providing faster training, lesser training errors, and overfitting.

Furthermore, the analog router fuses well with analog compute-in-memory (CiM) tiles and reduces data conversions between the tiles, resulting in lower-power designs relative to conventional solutions. At the higher level, the NoC provides the flexibility of a digital implementation with packet switched network for data movement to nodes farther away where analog solutions tend to underperform, as loss may occur when transmitting analog signals over distances. Doing so may generally improve system performance by reducing the amount of time and/or energy required to perform analog-to-digital conversions and digital-to-analog conversions. Furthermore, by providing a configurable neural network architecture, the NoC is adaptable to any type of neural network implementations. More generally, by reducing the number of analog/digital conversions, energy and system resources are conserved, which improves the time and amount of resources required to train neural networks and/or perform inference operations using the neural networks.

With general reference to notations and nomenclature used herein, one or more portions of the detailed description which follows may be presented in terms of program procedures executed on a computer or network of computers. These procedural descriptions and representations are used by those skilled in the art to effectively convey the substances of their work to others skilled in the art. A procedure is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It proves convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. It should be noted, however, that these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to those quantities.

Further, these manipulations are often referred to in terms, such as adding or comparing, which are commonly associated with mental operations performed by a human operator. However, no such capability of a human operator is necessary, or desirable in most cases, in any of the operations described herein that form part of one or more embodiments. Rather, these operations are machine operations. Useful machines for performing operations of various embodiments include general purpose digital computers as selectively activated or configured by a computer program stored within that is written in accordance with the teachings herein, and/or include apparatus specially constructed for the required purpose. Various embodiments also relate to apparatus or systems for performing these operations. These apparatuses may be specially constructed for the required purpose or may include a general-purpose computer. The required structure for a variety of these machines will be apparent from the description given.

Reference is now made to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding thereof. It may be evident, however, that the novel embodiments can be practiced without these specific details. In other instances, well known structures and devices are shown in block diagram form to facilitate a description thereof. The intention is to cover all modification, equivalents, and alternatives within the scope of the claims.

In the Figures and the accompanying description, the designations “a” and “b” and “c” (and similar designators) are intended to be variables representing any positive integer. Thus, for example, if an implementation sets a value for a=5, then a complete set of components122illustrated as components122-1through122-amay include components122-1,122-2,122-3,122-4, and122-5. The embodiments are not limited in this context.

FIG.1illustrates a system100for hierarchical hybrid network on a chip architecture for compute-in-memory probabilistic machine learning accelerators, according to one embodiment. As shown, the system100comprises a network on a chip102. The NoC102may be any suitable logic for training a neural network and/or executing a trained neural network, e.g., to perform runtime (or inference) operations. For example, the NoC102may be implemented as a deep learning accelerator card, a processor with deep learning acceleration, a neural compute stick, a System on a Chip (SoC), a printed circuit board, or a chip package with two or more discrete components. Embodiments are not limited in these contexts.

As shown, the network on a chip102includes a plurality supertiles106-1through106-N coupled via a digital communications bus110. The bus110may include a plurality of digital routers126-1through126-N to facilitate digital packet switching, transmission, arbitration, and other communication operations for the network on a chip102. The bus110and digital routers126may generally form the digital backbone of the supertiles106, while the supertiles106operate in the analog domain. Although depicted as a mesh configuration, the bus110may be implemented according to other topologies, such as a tree-based topology, and the like. As shown in supertile106-1, each supertile106includes an analog router104and a plurality of compute-in-memory tiles108. Therefore, four analog routers104, one from each supertile106, may interface with a given digital router126of the NoC102.

Generally, each of the compute-in-memory tiles108is configured to perform compute-in-memory operations in the analog domain using a processor112and a memory114. The processors112are representative of any type of processing circuitry, and the memory114is representative of any type of information storage technology, including volatile technologies requiring the uninterrupted provision of electric power, and including technologies entailing the use of machine-readable storage media that may or may not be removable. In at least one embodiment, the processor112includes logic to perform multiply-accumulate (MAC) operations or other computations for a neural network. A MAC operation may generally compute the product of two numbers and add the product to an accumulator. Illustratively, each supertile106includes four compute-in-memory tiles108, but any number of compute-in-memory tiles may be included in a given supertile106.

The analog router104includes logic116which is generally configured to facilitate information exchange between the compute-in-memory tiles108of the supertile106and/or the compute-in-memory tiles108of other supertiles106. The logic116is representative of any type of processing circuitry. The analog router104may further include a neural network configuration118, a Gaussian logic circuit120, one or more analog-to-digital converters (ADCs)122, and one or more digital-to-analog converters (DACs)120. The neural network configuration118includes all parameters to implement a neural network on the NoC102. For example, the neural network configuration118may specify the number of layers, how the layers of the neural network are connected, weights of each node in the neural network, and the like. Example neural networks include, but are not limited to, Deep Neural Networks (DNNs) such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and the like. A neural network generally implements dynamic programing to determine and solve for an approximated value function. A neural network is formed of a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Generally, each successive layer of a neural network uses the output from the previous layer as input. A neural network may generally include an input layer, an output layer, and multiple hidden layers. The hidden layers of a neural network may include convolutional layers, pooling layers, fully connected layers, SoftMax layers, and/or normalization layers. In one embodiment, the plurality of hidden layers comprise three hidden layers (e.g., a count of the hidden layers comprises three hidden layers). Embodiments are not limited in these contexts.

Generally, a neural network includes two processing phases, a training phase and an inference phase. During the training process, a deep learning expert will typically architect the network, establishing the number of layers in the network, the operation performed by each layer, and the connectivity between layers. Many layers have parameters, which may be referred to as weights, that determine exact computation performed by the layer. The objective of the training process is to learn the weights, usually via a stochastic gradient descent-based excursion through the space of weights. Once the training process is complete, inference based on the trained neural network (e.g., image analysis, image and/or video encoding, image and/or video decoding, face detection, character recognition, speech recognition, etc.) typically employs a forward-propagation calculation for input data to generate output data.

The analog router104may generally perform conversions between the analog and digital domains (and vice versa) using the DACs122and the ADCs124. Advantageously, however, the analog router104may limit the number of conversions based on the neural network configuration118. Generally, when transmitting data within a supertile106, the analog router104may refrain from converting the data from the analog domain to the digital domain and back to the analog domain. For example, the neural network configuration118may specify that the analog output of a compute-in-memory tile108aof a supertile106ais to be provided to a compute-in-memory tile108bof supertile106aas input. Therefore, based on the neural network configuration118, the analog router104may refrain from converting the analog output to digital, and instead directly provide the analog signal to the compute-in-memory tile108bof the supertile106afor processing.

However, if the data is to be transmitted between different supertiles106, the ADC124of the transmitting supertile106may convert the analog data to the digital domain (e.g., by packetizing the analog data into one or more data packets), and the receiving analog router104may convert the digital signal (e.g., one or more data packets) into an analog signal using the DAC122. The digital routers126may generally facilitate the transfer of the packets between supertiles106of the network on a chip102using the digital router logic128. In some embodiments, the digital router logic128includes the neural network configuration118, which may be used by the digital router logic128to route packets via the bus110. Doing so facilitates a flexible configuration whereby different configurations of neural networks can be processed by the NoC102.

Advantageously, the analog routers104provide energy savings on the network on a chip102by avoiding data conversion overhead within a supertile106, while the digital routers126facilitate flexible and scalable solutions, as described in greater detail herein. Doing so improves the performance of the network on a chip102by requiring less energy to power the network on a chip102relative to a network on a chip102that performs analog to digital conversions (and vice versa) in all scenarios. Furthermore, because analog signals are more vulnerable to interference during transmission, e.g., path loss, noise coupling, and ambient environment variations may significantly impact the fidelity of the analog signal. Interference shields and signal strength boosters may assist, but consume on-chip area and energy. Advantageously, by limiting the transmission of analog signals to within a supertile106, interference shields and signal strength boosters may be omitted from the configuration of the network on a chip102.

The Gaussian logic120is circuitry embedded in the analog router104for supporting probabilistic machine learning applications. Generally, the Gaussian logic120may convert the inputs, outputs, and/or weights of a deterministic neural network into a Bayesian neural network by including a programmable noise source. For example, a neuron (also referred to as a node) of a neural network may produce a value as output. However, in probabilistic settings, a distribution, or range of a plurality of values, may be desired. Therefore, the Gaussian logic120may generally generate the distribution of values, which may include a mean value and a variance (and/or standard deviation). The distribution may comprise a Gaussian (or normal) probability distribution. Advantageously, the Gaussian logic120performs the computations in the analog domain, which is more efficient than the digital domain. The Gaussian logic120may introduce scaled Gaussian noise in one or more of: the inputs to the neural network, the weights of the neural network, and/or the outputs of a layer of the neural network as they pass through an analog router104. Advantageously, introducing the noise at the output of one layer of the neural network (or the input of a layer of the neural network) may provide the solution having the highest throughput and least energy use requirements.

Furthermore, the Gaussian logic120may be configured to perform stochastic rounding operations in the analog domain. For example, the stochastic rounding operation may occur between layers of the neural network. Stochastic rounding operations may result in higher accuracy in reduced-precision neural networks, which may be akin to increasing the bit width of the neural network. Furthermore, when used in backpropagation operations in neural network training, stochastic rounding results in faster training of the neural network, lower training error, and less problems of overfitting the training dataset. Performing the stochastic rounding operation in the analog domain incurs almost no overhead in the analog router104, and less overhead than performing stochastic rounding operations in the digital domain. Advantageously, providing the Gaussian logic120in the analog router104obviates the need to include an instance of the Gaussian logic120in each compute-in-memory tile108.

FIG.2Aillustrates a portion of a supertile106a, according to one embodiment. As shown, the supertile106aincludes four compute-in-memory tiles108a-108dcoupled by example connections202a-202c(among other connections not labeled for the sake of clarity). As shown, the compute-in-memory tiles108a-108dincludes analog processing element (PE)204a-d, an analog input206a-d, a digital bit line208a-d, a word line210a-d, and analog output212a-d. Generally, inFIGS.2A-2D, a (D) indicates a hardware element that processes and/or stores data in the digital domain, while an (A) indicates a hardware element that processes and/or stores data in the analog domain. The analog PEs204a-dmay be representative of the processor112and memory114ofFIG.1.

The connections202may generally be selected based on the neural network configuration118, e.g., the topology of the neural network. Furthermore, the neural network configuration118may specify that the output212a-dshould be provided as input to another supertile106. As stated, the analog routers104(not explicitly labeled inFIGS.2A-2Dfor clarity) may not convert analog output212a-dto digital when transmitting data within the supertile106. For example, if the output (A)212cis to be provided to input (A)206d, the logic116of the analog router104may refrain from converting the output (A)212cto digital packets. However, if the output (A)212cis to be transmitted to a different supertile106b, the analog router104may convert the analog output212cto one or more packets using the ADC124, and transmit the packets to supertile106bvia the one or more digital routers126of the bus110. The receiving supertile106bmay use the DAC122to convert the packets to one or more analog signals for processing by the compute-in-memory tiles108of supertile106b.

FIG.2Bdepicts different example connections202a-202c, according to various embodiments. As shown, connection202amay reflect a scenario where analog signals may experience high impedance at the input of the next stage. In such embodiments, a passive connection, such as a piece of wire, is suitable for transmitting analog signals. If the next stage is low impedance and/or requires driven capabilities, the connection may include an active signal relay. The relay may convert the signal with high fidelity (e.g., linearly) or pass the signal with distortion (nonlinear). The connection202binFIG.2Bmay be a linear relay, while the connection202cinFIG.2Bmay be nonlinear. The connection202bdepicted inFIG.2Bmay require more hardware and/or energy consumption relative to the connection202c. As shown, each connection202a-cmay include one or more multiplexers for signal selection at the input and output of each compute-in-memory tile108.

FIG.2Cillustrates an embodiment reflecting an example configuration of a supertile106based on a neural network configuration118. As stated, a neural network may be mapped across the network on a chip102based on the neural network configuration118. A given supertile106and the compute-in-memory tiles108of the supertile106may generally perform one or more tasks of the neural network, with other supertiles106and corresponding compute-in-memory tiles108performing other tasks for the neural network. When multiple layers of a neural network are processed within a single supertile106, as depicted inFIG.2C, the computation is distributed across the compute-in-memory tiles108a-d, and the analog router104manages the transfer of data between the compute-in-memory tiles108a-d, as well as the transfer of data to other supertiles106.

More specifically, as shown inFIG.2C, the supertile106aprocesses three layers of the neural network, namely layers1,2, and3. However, as shown, layers1and3are processed by analog PE204aand analog PE204cof compute-in-memory tiles108aand108c, respectively. Furthermore, layer2requires two layers, layers2.1and2.2, which are processed by analog PE204band analog PE204dof compute-in-memory tiles108band108d, respectively. A portion of the analog data outputted by compute-in-memory tile108ais transmitted to compute-in-memory tile108bvia connection202awithout requiring conversion from analog to digital, and back from digital to analog. Similarly, a portion of the analog output of compute-in-memory tile108ais transmitted to compute-in-memory tile108dvia connection202b. Similarly, the analog output of compute-in-memory tile108bis transmitted to compute-in-memory tile108cvia connection202d, while the analog output of compute-in-memory tile108dis transmitted to compute-in-memory tile108cvia connection202e. Advantageously, all data transmission depicted inFIG.2Cdo not require conversion from analog to digital, and back from digital to analog.

FIG.2Ddepicts an example configuration of a supertile106based on a neural network configuration118, where the configuration is different than that ofFIG.2C. As shown, the supertile106aprocesses two layers of the neural network, namely layers1, and2. Layer1is processed by analog PE204aof compute-in-memory tile108a, while layer2includes three sub-layers, namely layers2.1,2.2, and2.3. As shown, layer2.1is processed by analog PE204bof compute-in-memory tile108b, layer2.2is processed by analog PE204dof compute-in-memory tile108d, and layer2.3is processed by analog PE204cof compute-in-memory tile108c. The analog router104may include connections202a-cfor transmitting analog output of compute-in-memory tiles108ato compute-in-memory tiles108b-d. Advantageously,FIGS.2C-2Dreflect the flexibility of supporting different neural network configurations118, where the different supertiles106support one of a plurality of possible configurations to realize a larger neural network.

FIG.3depicts an example configuration of the Gaussian logic120, also referred to as a Gaussian distribution circuit, according to one embodiment. As stated, the generation of a Gaussian distribution is expensive to implement in digital hardware in terms of power, area on the NoC102, and throughput. Analog signal transmission offers a less complex alternative option to introduce uncertainty into a static signal along with the data transmission for use in probabilistic computing by the neural network.

As shown, the Gaussian logic120includes a sample and hold circuit301, an entropy source302, a dithering charge pump303, and a capacitor306. The sample and hold circuit301tracks the output of the compute-in-memory tiles108and converts the signal into a charge (or voltage) stored on the positive plate of the capacitor306. After charging the capacitor306, the value on the node is the mean μ of the distribution. In one example, the charge (or voltage) stored on the positive plate may be the mean μ of the distribution. The entropy source302may comprise a ring oscillator304, also referred to as a jittery oscillator. By using the jittery signal of the oscillator304, to sample a relatively jitter-free reference clock305signal (e.g., a crystal oscillator) around its expected transition, the Gaussian distributed randomness in the time domain is transferred into the voltage domain by generating an output voltage in each cycle. The adjustment ability of the distribution variance σ2is obtained by tuning the slope of the reference clock305. A steeper transition of the reference clock signal results in a wider sampling window and therefore leads to a large variation. Similarly, a flatter transition of the reference clock signal results in a narrower sampling window and leads towards a lesser variation.

The Gaussian logic120may then apply the distribution variance σ2onto the previously obtained amount of charge using a charge pump303modulated by the entropy source302. The amount of charge at the node is modulated by the charge pump303in two adjacent clock cycles to discharge and recharge the value. The slope of the reference clock changes every two cycles based on the variance σ2. In the first cycle, the output node is discharged with a current volume determined by the sampled voltage value VN. In the second cycle, it is recharged by the volume controlled by the sampled voltage value VN+1at the second time step. The distribution variance σ2may then be determined as the difference between VNand VN+1. By implementing this approach in the analog domain, the implementation of stochastic rounding operations is straightforward, e.g., by setting the variance σ2to the level of a single least significant bit (LSB).

Operations for the disclosed embodiments may be further described with reference to the following figures. Some of the figures may include a logic flow. Although such figures presented herein may include a particular logic flow, it can be appreciated that the logic flow merely provides an example of how the general functionality as described herein can be implemented. Further, a given logic flow does not necessarily have to be executed in the order presented unless otherwise indicated. Moreover, not all acts illustrated in a logic flow may be required in some embodiments. In addition, the given logic flow may be implemented by a hardware element, a software element executed by a processor, or any combination thereof. The embodiments are not limited in this context.

FIG.4illustrates an embodiment of a logic flow, or routine,400. The logic flow400may be representative of some or all of the operations executed by one or more embodiments described herein. For example, the logic flow400may include some or all of the operations to use an analog router to perform conversions of data between the analog domain and the digital domain in a network on a chip that implements a neural network using compute-in-memory logic. Embodiments are not limited in this context.

In block402, routine400receives, by an analog router104of a first supertile106aof a plurality of supertiles106a-106nof a network on a chip (NoC)102, a first analog output from a first compute-in-memory tile108aof a plurality of compute-in-memory tiles108a-108nof the first supertile106. In block404, routine400determines, by the analog router104based on a configuration of a neural network executing on the NoC, a destination of the first analog output comprises a second supertile106bof the plurality of supertiles. In block406, a Gaussian distribution circuit120of the analog router104generates a distribution comprising a plurality of weights for the weight of the neural network, and optionally performs a stochastic rounding operation. The weight may be included in the first analog output received at block402. The weight may be for a node (or neuron) of the neural network, a layer of the neural network, or any other suitable component of the neural network. In block408, an analog-to-digital converter (ADC)124of the analog router104converts the first analog output and the Gaussian distribution to a first digital output, e.g., one or more digital data packets. In block410, the analog router104transmits the one or more packets including first digital output and the distribution to the second supertile via a communications bus110of the NoC102. The bus110may include one or more digital routers126between the first and second supertiles that facilitate routing of the packets.

FIG.5illustrates an embodiment of a logic flow, or routine,500. The logic flow500may be representative of some or all of the operations executed by one or more embodiments described herein. For example, the logic flow500may include some or all of the operations to use an analog router to perform conversions of data between the analog domain and the digital domain in a network on a chip that implements a neural network using compute-in-memory logic. Embodiments are not limited in this context.

In block502, routine500receives, by an analog router104of a first supertile106aof a plurality of supertiles106a-106nof a NoC102, a first analog output from a first compute-in-memory tile108aof a plurality of compute-in-memory tiles108a-108nof the first supertile106a. In block504, routine500determines, by the analog router104based on a configuration of a neural network executing on the NoC, a destination of the first analog output comprises a second compute-in-memory tile108bof the plurality of compute-in-memory tiles108a-108nof the first supertile106. In block506, routine500generates, by a Gaussian distribution circuit120of the analog router104, a distribution comprising a plurality of weights for the weight of the neural network. The weight may be included in the first analog output received at block502. The weight may be for a node (or neuron) of the neural network, a layer of the neural network, or any other suitable component of the neural network. In block508, routine500transmits, by the analog router, the first analog output and the distribution to the second compute-in-memory tile without converting the second analog output to the digital domain. Advantageously, doing so reduces the amount of energy used by the NoC102, as the NoC102need not convert the analog output and the distribution from the analog domain to the digital domain, and vice versa.

FIG.6illustrates an embodiment of a logic flow, or routine,600. The logic flow600may be representative of some or all of the operations executed by one or more embodiments described herein. For example, the logic flow600may include some or all of the operations to use an analog router to perform conversions of data between the analog domain and the digital domain in a network on a chip that implements a neural network using compute-in-memory logic. Embodiments are not limited in this context.

In block602, routine600receives, by an analog router104of a first supertile106aof a plurality of supertiles106a-106nof a NoC102, a digital packet comprising a weight of a neural network from a second supertile106bof the plurality of supertiles106a-106n. The packet may be received via the bus110and one or more digital routers126between the supertiles106aand106b. In block604, routine600converts, by a digital-to-analog converter (DAC)122of the analog router104of the first supertile106a, the digital packet to an analog signal. In block606, routine600generates, by a Gaussian distribution circuit120of the analog router104, a distribution comprising a plurality of weights for the weight. In block608, routine600provides, by the analog router, the analog signal and the distribution to one of a plurality of compute-in-memory tiles108a-108nof the first supertile106a.

FIG.7illustrates an embodiment of a storage medium702. Storage medium702may comprise any non-transitory computer-readable storage medium or machine-readable storage medium, such as an optical, magnetic or semiconductor storage medium. In various embodiments, storage medium702may comprise an article of manufacture. In some embodiments, storage medium702may store computer-executable instructions, such as computer-executable instructions to implement one or more of logic flows or operations described herein, such as computer-executable instructions704for logic flow400ofFIG.4, computer-executable instructions704for logic flow500ofFIG.5, and computer-executable instructions706for logic flow600ofFIG.6. Examples of a computer-readable storage medium or machine-readable storage medium may include any tangible media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. Examples of computer-executable instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, and the like. The embodiments are not limited in this context.

FIG.8illustrates an embodiment of a system800. System800is a computer system with multiple processor cores such as a distributed computing system, supercomputer, high-performance computing system, computing cluster, mainframe computer, mini-computer, client-server system, personal computer (PC), workstation, server, portable computer, laptop computer, tablet computer, handheld device such as a personal digital assistant (PDA), or other device for processing, displaying, or transmitting information. Similar embodiments may comprise, e.g., entertainment devices such as a portable music player or a portable video player, a smart phone or other cellular phone, a telephone, a digital video camera, a digital still camera, an external storage device, or the like. Further embodiments implement larger scale server configurations. In other embodiments, the system800may have a single processor with one core or more than one processor. Note that the term “processor” refers to a processor with a single core or a processor package with multiple processor cores. In at least one embodiment, the computing system800is representative of the network on a chip102of the system100. More generally, the computing system800is configured to implement all logic, systems, logic flows, methods, apparatuses, and functionality described herein with reference toFIGS.1-7.

As used in this application, the terms “system” and “component” and “module” are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution, examples of which are provided by the exemplary system800. For example, a component can be, but is not limited to being, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical and/or magnetic storage medium), an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and/or thread of execution, and a component can be localized on one computer and/or distributed between two or more computers. Further, components may be communicatively coupled to each other by various types of communications media to coordinate operations. The coordination may involve the uni-directional or bi-directional exchange of information. For instance, the components may communicate information in the form of signals communicated over the communications media. The information can be implemented as signals allocated to various signal lines. In such allocations, each message is a signal. Further embodiments, however, may alternatively employ data messages. Such data messages may be sent across various connections. Exemplary connections include parallel interfaces, serial interfaces, and bus interfaces.

As shown in this figure, system800comprises a motherboard or system-on-chip (SoC)802for mounting platform components. The motherboard or system-on-chip (SoC)802may be representative of the NoC102ofFIG.1. Motherboard or system-on-chip (SoC)802is a point-to-point (P2P) interconnect platform that includes a first processor804and a second processor806coupled via a point-to-point interconnect870such as an Ultra Path Interconnect (UPI). In other embodiments, the system800may be of another bus architecture, such as a multi-drop bus. Furthermore, each of processor804and processor806may be processor packages with multiple processor cores including core(s)808and core(s)810, respectively. While the system800is an example of a two-socket (2S) platform, other embodiments may include more than two sockets or one socket. For example, some embodiments may include a four-socket (4S) platform or an eight-socket (8S) platform. Each socket is a mount for a processor and may have a socket identifier. Note that the term platform refers to the motherboard with certain components mounted such as the processor804and chipset832. Some platforms may include additional components and some platforms may only include sockets to mount the processors and/or the chipset. Furthermore, some platforms may not have sockets (e.g. SoC, or the like).

The processor804and processor806can be any of various commercially available processors, including without limitation an Intel® Celeron®, Core®, Core (2) Duo®, Itanium®, Pentium®, Xeon®, and XScale® processors; AMD® Athlon®, Duron® and Opteron® processors; ARM® application, embedded and secure processors; IBM® and Motorola® DragonBall® and PowerPC® processors; IBM and Sony® Cell processors; and similar processors. Dual microprocessors, multi-core processors, and other multi-processor architectures may also be employed as the processor804and/or processor806. Additionally, the processor804need not be identical to processor806.

Processor804includes an integrated memory controller (IMC)820and point-to-point (P2P) interface824and P2P interface828. Similarly, the processor806includes an IMC822as well as P2P interface826and P2P interface830. IMC820and IMC822couple the processors processor804and processor806, respectively, to respective memories (e.g., memory816and memory818). Memory816and memory818may be portions of the main memory (e.g., a dynamic random-access memory (DRAM)) for the platform such as double data rate type3(DDR3) or type4(DDR4) synchronous DRAM (SDRAM). In the present embodiment, the memories memory816and memory818locally attach to the respective processors (i.e., processor804and processor806). In other embodiments, the main memory may couple with the processors via a bus and shared memory hub.

System800includes chipset832coupled to processor804and processor806. Furthermore, chipset832can be coupled to storage device850, for example, via an interface (I/F)838. The I/F838may be, for example, a Peripheral Component Interconnect-enhanced (PCI-e). Storage device850can store instructions executable by circuitry of system800(e.g., processor804, processor806, GPU848, ML accelerator854, vision processing unit856, or the like).

Processor804couples to a chipset832via P2P interface828and P2P834while processor806couples to a chipset832via P2P interface830and P2P836. Direct media interface (DMI)876and DMI878may couple the P2P interface828and the P2P834and the P2P interface830and P2P836, respectively. DMI876and DMI878may be a high-speed interconnect that facilitates, e.g., eight Giga Transfers per second (GT/s) such as DMI 3.0. In other embodiments, the processor804and processor806may interconnect via a bus.

The chipset832may comprise a controller hub such as a platform controller hub (PCH). The chipset832may include a system clock to perform clocking functions and include interfaces for an I/O bus such as a universal serial bus (USB), peripheral component interconnects (PCIs), serial peripheral interconnects (SPIs), integrated interconnects (I2Cs), and the like, to facilitate connection of peripheral devices on the platform. In other embodiments, the chipset832may comprise more than one controller hub such as a chipset with a memory controller hub, a graphics controller hub, and an input/output (I/O) controller hub.

In the depicted example, chipset832couples with a trusted platform module (TPM)844and UEFI, BIOS, FLASH circuitry846via I/F842. The TPM844is a dedicated microcontroller designed to secure hardware by integrating cryptographic keys into devices. The UEFI, BIOS, FLASH circuitry846may provide pre-boot code.

Furthermore, chipset832includes the I/F838to couple chipset832with a high-performance graphics engine, such as, graphics processing circuitry or a graphics processing unit (GPU)848. In other embodiments, the system800may include a flexible display interface (FDI) (not shown) between the processor804and/or the processor806and the chipset832. The FDI interconnects a graphics processor core in one or more of processor804and/or processor806with the chipset832.

Additionally, ML accelerator854and/or vision processing unit856can be coupled to chipset832via I/F838. ML accelerator854can be circuitry arranged to execute ML related operations (e.g., training, inference, etc.) for ML models. Likewise, vision processing unit856can be circuitry arranged to execute vision processing specific or related operations. In particular, ML accelerator854and/or vision processing unit856can be arranged to execute mathematical operations and/or operands useful for machine learning, neural network processing, artificial intelligence, vision processing, etc.

Various I/O devices860and display852couple to the bus872, along with a bus bridge858which couples the bus872to a second bus874and an I/F840that connects the bus872with the chipset832. In one embodiment, the second bus874may be a low pin count (LPC) bus. Various devices may couple to the second bus874including, for example, a keyboard862, a mouse864and communication devices866.

Furthermore, an audio I/O868may couple to second bus874. Many of the I/O devices860and communication devices866may reside on the motherboard or system-on-chip (SoC)802while the keyboard862and the mouse864may be add-on peripherals. In other embodiments, some or all the I/O devices860and communication devices866are add-on peripherals and do not reside on the motherboard or system-on-chip (SoC)802.

One or more aspects of at least one example may be implemented by representative instructions stored on at least one machine-readable medium which represents various logic within the processor, which when read by a machine, computing device or system causes the machine, computing device or system to fabricate logic to perform the techniques described herein. Such representations, known as “IP cores” may be stored on a tangible, machine readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that make the logic or processor.

Various examples may be implemented using hardware elements, software elements, or a combination of both. In some examples, hardware elements may include devices, components, processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), memory units, logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. In some examples, software elements may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an example is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints, as desired for a given implementation.

Some examples may include an article of manufacture or at least one computer-readable medium. A computer-readable medium may include a non-transitory storage medium to store logic. In some examples, the non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. In some examples, the logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, API, instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof.

According to some examples, a computer-readable medium may include a non-transitory storage medium to store or maintain instructions that when executed by a machine, computing device or system, cause the machine, computing device or system to perform methods and/or operations in accordance with the described examples. The instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, and the like. The instructions may be implemented according to a predefined computer language, manner or syntax, for instructing a machine, computing device or system to perform a certain function. The instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled and/or interpreted programming language.

Some examples may be described using the expression “in one example” or “an example” along with their derivatives. These terms mean that a particular feature, structure, or characteristic described in connection with the example is included in at least one example. The appearances of the phrase “in one example” in various places in the specification are not necessarily all referring to the same example.

Some examples may be described using the expression “coupled” and “connected” along with their derivatives. These terms are not necessarily intended as synonyms for each other. For example, descriptions using the terms “connected” and/or “coupled” may indicate that two or more elements are in direct physical or electrical contact with each other. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, yet still co-operate or interact with each other.

The following examples pertain to further embodiments, from which numerous permutations and configurations will be apparent.

Example 1 is a method, comprising: receiving, by an analog router of a first supertile of a plurality of supertiles of a network on a chip (NoC), a first analog output from a first compute-in-memory tile of a plurality of compute-in-memory tiles of the first supertile; determining, by the analog router based on a configuration of a neural network executing on the NoC, that a destination of the first analog output comprises a second supertile of the plurality of supertiles; converting, by an analog-to-digital converter (ADC) of the analog router, the first analog output to a first digital output; and transmitting, by the analog router, the first digital output to the second supertile via a communications bus of the NoC.

Example 2 includes the subject matter of example 1, wherein the first analog output comprises a weight of the neural network, further comprising: generating, by a Gaussian distribution circuit of the analog router, a Gaussian distribution comprising a mean and a variance of the weight; and transmitting, by the analog router to the second supertile, the distribution and the first digital output via at least one digital router of a plurality of digital routers of the communications bus.

Example 3 includes the subject matter of example 2, further comprising: performing, by the Gaussian distribution circuit, a stochastic rounding operation on the weight.

Example 4 includes the subject matter of example 1, further comprising: receiving, by the analog router, a second analog output of the first compute-in-memory tile; determining, by the analog router, that a destination of the second analog output comprises a second compute-in-memory tile of the plurality of compute-in-memory tiles of the first supertile; and transmitting, by the analog router, the second analog output to the second compute-in-memory tile without converting the second analog output.

Example 5 includes the subject matter of example 4, wherein the output comprises a weight of the neural network, further comprising: generating, by a Gaussian distribution circuit of the analog router, a Gaussian distribution comprising a plurality of weights for the weight; and transmitting, by the analog router to the second compute-in-memory tile, the distribution and the second analog output.

Example 6 includes the subject matter of example 1, further comprising: receiving, by the analog router via a digital router of the communications bus, a digital packet from the second supertile; converting, by a digital-to-analog converter (DAC) of the analog router, the digital packet to an analog signal; and providing, by the analog router, the analog signal to one of the compute-in-memory tiles of the first supertile.

Example 7 includes the subject matter of example 1, wherein each compute-in-memory tile comprises a respective processing circuitry and a respective memory, wherein each supertile comprises a respective analog router and a respective plurality of compute-in-memory tiles, the method further comprising: packetizing, by the analog router, the first digital output to one or more packets prior to transmitting the first digital output to the second supertile.

Example is 8 an apparatus for a network on a chip (NoC), comprising: a plurality of supertiles, each supertile comprising an analog router and a plurality of compute-in-memory tiles; and logic, at least a portion of which is implemented in the analog router of a first supertile of the plurality of supertiles, the logic to: receive a first analog output from a first compute-in-memory tile of a plurality of compute-in-memory tiles of the first supertile; determine, based on a configuration of a neural network executing on the NoC, that a destination of the first analog output comprises a second supertile of the plurality of supertiles; convert, by an analog-to-digital converter (ADC) of the analog router, the first analog output to a first digital output; and transmit, by the analog router, the first digital output to the second supertile via a communications bus of the NoC.

Example 9 includes the subject matter of example 8, wherein the first analog output comprises a weight of the neural network, wherein the logic is further configured to: generate, by a Gaussian distribution circuit of the analog router, a Gaussian distribution comprising a plurality of weights for the weight; and transmit the distribution and the first digital output to the second supertile via at least one digital router of a plurality of digital routers of the NoC.

Example 10 includes the subject matter of example 9, wherein the logic is further configured to: perform, by the Gaussian distribution circuit, a stochastic rounding operation on the weight.

Example 11 includes the subject matter of example 8, wherein the logic is further configured to: receive, by the analog router, a second analog output of the first compute-in-memory tile; determine, by the analog router, that a destination of the second analog output comprises a second compute-in-memory tile of the plurality of compute-in-memory tiles of the first supertile; and transmit, by the analog router, the second analog output to the second compute-in-memory tile without converting the second analog output.

Example 12 includes the subject matter of example 11, wherein the output comprises a weight of the neural network, wherein the logic is further configured to: generate, by a Gaussian distribution circuit of the analog router, a Gaussian distribution comprising a plurality of weights for the weight; and transmit, by the analog router to the second compute-in-memory tile, the distribution and the second analog output.

Example 13 includes the subject matter of example 8, wherein the logic is further configured to: receive, by the analog router via a digital router of the communications bus, a digital packet from the second supertile; convert, by a digital-to-analog converter (DAC) of the analog router, the digital packet to an analog signal; and provide, by the analog router, the analog signal to one of the compute-in-memory tiles of the first supertile.

Example 14 includes the subject matter of example 8, wherein each compute-in-memory tile comprises a respective processing circuitry and a respective memory, wherein each supertile comprises a respective analog router and a respective plurality of compute-in-memory tiles, wherein the logic is further configured to: packetize the first digital output to one or more packets prior to transmitting the first digital output to the second supertile.

Example 15 is a non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a network on a chip, cause the NoC to: receive, by an analog router of a first supertile of a plurality of supertiles of the NoC, a first analog output from a first compute-in-memory tile of a plurality of compute-in-memory tiles of the first supertile; determine, by the analog router based on a configuration of a neural network executing on the NoC, that a destination of the first analog output comprises a second supertile of the plurality of supertiles; convert, by an analog-to-digital converter (ADC) of the analog router, the first analog output to a first digital output; and transmit, by the analog router, the first digital output to the second supertile via a communications bus of the NoC.

Example 16 includes the subject matter of example 15, wherein the first analog output comprises a weight of the neural network, wherein the instructions further configure the computer to: generate, by a Gaussian distribution circuit of the analog router, a Gaussian distribution comprising a plurality of weights for the weight; and transmit, by the analog router to the second supertile, the distribution and the first digital output via at least one digital router of a plurality of digital routers of the communications bus.

Example 17 includes the subject matter of example 16, wherein the instructions further configure the NoC to: perform, by the Gaussian distribution circuit, a stochastic rounding operation on the weight. 18 includes the subject matter of example 15, wherein the instructions further configure the NoC to: receive, by the analog router, a second analog output of the first compute-in-memory tile; determine, by the analog router, that a destination of the second analog output comprises a second compute-in-memory tile of the plurality of compute-in-memory tiles of the first supertile; and transmit, by the analog router, the second analog output to the second compute-in-memory tile without converting the second analog output.

Example 19 includes the subject matter of example 18, wherein the output comprises a weight of the neural network, wherein the instructions further configure the NoC to: generate, by a Gaussian distribution circuit of the analog router, a Gaussian distribution comprising a plurality of weights for the weight; and transmit, by the analog router to the second compute-in-memory tile, the distribution and the second analog output via at least one digital router of a plurality of digital routers of the NoC.

Example 20 includes the subject matter of example 15, wherein each compute-in-memory tile comprises a respective processing circuitry and a respective memory, wherein each supertile comprises a respective analog router and a respective plurality of compute-in-memory tiles, wherein the instructions further configure the NoC to: receive, by the analog router via a digital router of the communications bus, a digital packet from the second supertile; convert, by a digital-to-analog converter (DAC) of the analog router, the digital packet to an analog signal; and provide, by the analog router, the analog signal to one of the compute-in-memory tiles of the first supertile.

Example 21 is an apparatus, comprising: means for receiving, by an analog router of a first supertile of a plurality of supertiles of a network on a chip (NoC), a first analog output from a first compute-in-memory tile of a plurality of compute-in-memory tiles of the first supertile; means for determining, by the analog router based on a configuration of a neural network executing on the NoC, that a destination of the first analog output comprises a second supertile of the plurality of supertiles; means for converting, by an analog-to-digital converter (ADC) of the analog router, the first analog output to a first digital output; and means for transmitting, by the analog router, the first digital output to the second supertile via a communications bus of the NoC.

Example 22 includes the subject matter of example 21, wherein the first analog output comprises a weight of the neural network, further comprising: means for generating, by a Gaussian distribution circuit of the analog router, a Gaussian distribution comprising a plurality of weights for the weight; and means for transmitting, by the analog router to the second supertile, the distribution and the first digital output via at least one digital router of a plurality of digital routers of the communications bus.

Example 23 includes the subject matter of example 22, further comprising: means for performing, by the Gaussian distribution circuit, a stochastic rounding operation on the weight.

Example 24 includes the subject matter of example 21, further comprising: means for receiving, by the analog router, a second analog output of the first compute-in-memory tile; means for determining, by the analog router, that a destination of the second analog output comprises a second compute-in-memory tile of the plurality of compute-in-memory tiles of the first supertile; and means for transmitting, by the analog router, the second analog output to the second compute-in-memory tile without converting the second analog output.

Example 25 includes the subject matter of example 24, wherein the output comprises a weight of the neural network, further comprising: means for generating, by a Gaussian distribution circuit of the analog router, a Gaussian distribution comprising a plurality of weights for the weight; and means for transmitting, by the analog router to the second compute-in-memory tile, the distribution and the second analog output.

Example 26 includes the subject matter of example 21, further comprising: means for receiving, by the analog router via a digital router of the communications bus, a digital packet from the second supertile; means for converting, by a digital-to-analog converter (DAC) of the analog router, the digital packet to an analog signal; and means for providing, by the analog router, the analog signal to one of the compute-in-memory tiles of the first supertile.

Example 27 includes the subject matter of example 21, wherein each compute-in-memory tile comprises a respective processing circuitry and a respective memory, wherein each supertile comprises a respective analog router and a respective plurality of compute-in-memory tiles, the apparatus further comprising: means for packetizing, by the analog router, the first digital output to one or more packets prior to transmitting the first digital output to the second supertile.

In addition, in the foregoing, various features are grouped together in a single example to streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate example. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein,” respectively. Moreover, the terms “first,” “second,” “third,” and so forth, are used merely as labels, and are not intended to impose numerical requirements on their objects.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

A data processing system suitable for storing and/or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code to reduce the number of times code must be retrieved from bulk storage during execution. The term “code” covers a broad range of software components and constructs, including applications, drivers, processes, routines, methods, modules, firmware, microcode, and subprograms. Thus, the term “code” may be used to refer to any collection of instructions which, when executed by a processing system, perform a desired operation or operations.

Logic circuitry, devices, and interfaces herein described may perform functions implemented in hardware and implemented with code executed on one or more processors. Logic circuitry refers to the hardware or the hardware and code that implements one or more logical functions. Circuitry is hardware and may refer to one or more circuits. Each circuit may perform a particular function. A circuit of the circuitry may comprise discrete electrical components interconnected with one or more conductors, an integrated circuit, a chip package, a chip set, memory, or the like. Integrated circuits include circuits created on a substrate such as a silicon wafer and may comprise components. And integrated circuits, processor packages, chip packages, and chipsets may comprise one or more processors.

Processors may receive signals such as instructions and/or data at the input(s) and process the signals to generate the at least one output. While executing code, the code changes the physical states and characteristics of transistors that make up a processor pipeline. The physical states of the transistors translate into logical bits of ones and zeros stored in registers within the processor. The processor can transfer the physical states of the transistors into registers and transfer the physical states of the transistors to another storage medium.

A processor may comprise circuits to perform one or more sub-functions implemented to perform the overall function of the processor. One example of a processor is a state machine or an application-specific integrated circuit (ASIC) that includes at least one input and at least one output. A state machine may manipulate the at least one input to generate the at least one output by performing a predetermined series of serial and/or parallel manipulations or transformations on the at least one input.

The logic as described above may be part of the design for an integrated circuit chip. The chip design is created in a graphical computer programming language and stored in a computer storage medium or data storage medium (such as a disk, tape, physical hard drive, or virtual hard drive such as in a storage access network). If the designer does not fabricate chips or the photolithographic masks used to fabricate chips, the designer transmits the resulting design by physical means (e.g., by providing a copy of the storage medium storing the design) or electronically (e.g., through the Internet) to such entities, directly or indirectly. The stored design is then converted into the appropriate format (e.g., GDSII) for the fabrication.

The resulting integrated circuit chips can be distributed by the fabricator in raw wafer form (that is, as a single wafer that has multiple unpackaged chips), as a bare die, or in a packaged form. In the latter case, the chip is mounted in a single chip package (such as a plastic carrier, with leads that are affixed to a motherboard or other higher level carrier) or in a multichip package (such as a ceramic carrier that has either or both surface interconnections or buried interconnections). In any case, the chip is then integrated with other chips, discrete circuit elements, and/or other signal processing devices as part of either (a) an intermediate product, such as a processor board, a server platform, or a motherboard, or (b) an end product.

The foregoing description of example embodiments has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Many modifications and variations are possible in light of this disclosure. It is intended that the scope of the present disclosure be limited not by this detailed description, but rather by the claims appended hereto. Future filed applications claiming priority to this application may claim the disclosed subject matter in a different manner and may generally include any set of one or more limitations as variously disclosed or otherwise demonstrated herein.