Object-Centric Learning with Slot Attention

A method involves receiving a perceptual representation including a plurality of feature vectors, and initializing a plurality of slot vectors represented by a neural network memory unit. Each respective slot vector is configured to represent a corresponding entity in the perceptual representation. The method also involves determining an attention matrix based on a product of the plurality of feature vectors transformed by a key function and the plurality of slot vectors transformed by a query function. Each respective value of a plurality of values along each respective dimension of the attention matrix is normalized with respect to the plurality of values. The method additionally involves determining an update matrix based on the plurality of feature vectors transformed by a value function and the attention matrix, and updating the plurality of slot vectors based on the update matrix by way of the neural network memory unit.

BACKGROUND

Machine Learning models may be used to process various types of data, including images, time series, text, and/or point clouds, among other possibilities. Improvements in the machine learning models allow the models to carry out the processing of data faster and/or utilize fewer computing resources for the processing.

SUMMARY

In a first example embodiment, a computer-implemented method is provided that includes receiving a perceptual representation that includes a plurality of feature vectors. The computer-implemented method also includes initializing a plurality of slot vectors represented by a neural network memory unit. Each respective slot vector of the plurality of slot vectors may be configured to represent a corresponding entity contained in the perceptual representation. The computer-implemented method additionally includes determining an attention matrix based on a product of (i) the plurality of feature vectors transformed by a key function and (ii) the plurality of slot vectors transformed by a query function. Each respective value of a plurality of values along each respective dimension of a plurality of dimensions of the attention matrix may be normalized with respect to the plurality of values along the respective dimension. The computer-implemented method further includes determining an update matrix based on (i) the plurality of feature vectors transformed by a value function and (ii) the attention matrix. The computer-implemented method yet further includes updating the plurality of slot vectors based on the update matrix by way of the neural network memory unit.

In a second example embodiment, a system is provided that includes a processor and a non-transitory computer-readable storage medium having stored thereon instruction that, when executed by the processor, cause the processor to perform operations. The operations include receiving a perceptual representation that includes a plurality of feature vectors. The operations also include initializing a plurality of slot vectors represented by a neural network memory unit. Each respective slot vector of the plurality of slot vectors may be configured to represent a corresponding entity contained in the perceptual representation. The operations additionally include determining an attention matrix based on a product of (i) the plurality of feature vectors transformed by a key function and (ii) the plurality of slot vectors transformed by a query function. Each respective value of a plurality of values along each respective dimension of a plurality of dimensions of the attention matrix may be normalized with respect to the plurality of values along the respective dimension. The operations further include determining an update matrix based on (i) the plurality of feature vectors transformed by a value function and (ii) the attention matrix. The operations yet further include updating the plurality of slot vectors based on the update matrix by way of the neural network memory unit.

In a third example embodiment, a non-transitory computer-readable storage medium is provided having stored thereon instruction that, when executed by a computing system, cause the computing system to perform operations. The operations include receiving a perceptual representation that includes a plurality of feature vectors. The operations also include initializing a plurality of slot vectors represented by a neural network memory unit. Each respective slot vector of the plurality of slot vectors may be configured to represent a corresponding entity contained in the perceptual representation. The operations additionally include determining an attention matrix based on a product of (i) the plurality of feature vectors transformed by a key function and (ii) the plurality of slot vectors transformed by a query function. Each respective value of a plurality of values along each respective dimension of a plurality of dimensions of the attention matrix may be normalized with respect to the plurality of values along the respective dimension. The operations further include determining an update matrix based on (i) the plurality of feature vectors transformed by a value function and (ii) the attention matrix. The operations yet further include updating the plurality of slot vectors based on the update matrix by way of the neural network memory unit.

In a fourth example embodiment, a system is provided that includes means for receiving a perceptual representation that includes a plurality of feature vectors. The system also includes means for initializing a plurality of slot vectors represented by a neural network memory unit. Each respective slot vector of the plurality of slot vectors may be configured to represent a corresponding entity contained in the perceptual representation. The system additionally includes means for determining an attention matrix based on a product of (i) the plurality of feature vectors transformed by a key function and (ii) the plurality of slot vectors transformed by a query function. Each respective value of a plurality of values along each respective dimension of a plurality of dimensions of the attention matrix may be normalized with respect to the plurality of values along the respective dimension. The system further includes means for determining an update matrix based on (i) the plurality of feature vectors transformed by a value function and (ii) the attention matrix. The system yet further includes means for updating the plurality of slot vectors based on the update matrix by way of the neural network memory unit.

DETAILED DESCRIPTION

Additionally, any enumeration of elements, blocks, or steps in this specification or the claims is for purposes of clarity. Thus, such enumeration should not be interpreted to require or imply that these elements, blocks, or steps adhere to a particular arrangement or are carried out in a particular order. Unless otherwise noted, figures are not drawn to scale.

A slot attention module may be configured to determine entity-centric (e.g., object-centric) representations of entities contained in a perceptual representation on the basis of a distributed representation of the perceptual representation. An example perceptual representation may take the form of an image that contains therein one or more entities such as objects, surfaces, regions, backgrounds, or other environmental features. Machine learning models may be configured to generate the distributed representation of the image. For example, one or more convolutional neural networks may be configured to process the image and generate one or more convolutional feature maps, which may represent the output of various feature filters implemented by the one or more convolutional neural networks.

These convolutional feature maps may be considered a distributed representation of the entities in the image because the features represented by the feature maps are related to different portions along the image area, but are not directly/explicitly associated with any of the entities represented in the image data. On the other hand, an object-centric representation may associate one or more features with individual entities represented in the image data. Thus, for example, each feature in a distributed representation may be associated with a corresponding portion of the perceptual representation, while each feature in an entity-centric representation may be associated with a corresponding entity contained in the perceptual representation.

Accordingly, the slot attention module may be configured to generate a plurality of entity-centric representations, referred to herein as slot vectors, based on a plurality of distributed representations, referred to herein as feature vectors. Each slot vector may be an entity-specific semantic embedding that represents the attributes or properties of one or more corresponding entities. The slot attention module may thus be considered to be an interface between perceptual representations and a structured set of variables represented by the slot vectors.

The slot attention module may include a neural network memory unit, such as a gated recurrent unit (GRU) or a long-short term memory (LSTM) neural network, configured to store and update the plurality of slot vectors across iterations of processing the feature vectors by the slot attention module. Parameters of the neural network memory unit may be learned during training of the slot attention module. The plurality of slot vectors may be initialized with values prior to a first iteration of processing by the slot attention module. In one example, the plurality of slot vectors may be initialized with random values selected, for example, from a normal distribution. In other implementations, the plurality of slot vectors may be initialized with values that cause one or more of the slot vectors to bind to and/or attach to particular entities (e.g., based on the values of previously-determined slot vectors). Thus, a particular slot vector may be caused to represent the same entity across successive perceptual representations (e.g., successive video frames or successive sections of audio waveforms) by initializing the particular slot vector with values of the particular slot vector previously determined with respect to one or more preceding perceptual representations.

The slot attention module may also include learned value, key, and query functions. The slot attention module may be configured to calculate an attention matrix based on a dot product of (i) the plurality of feature vectors transformed by the key function and (ii) the plurality of slot vectors transformed by the query function. Entries of the attention matrix may be normalized by way of a softmax function along a dimension corresponding to the slot vectors, thus causing the slot vectors to compete with one another for representing the entities contained in the perceptual representation. For example, when the number of slot vectors corresponds to a number of columns of the attention matrix, each value within each respective row of the attention matrix may be normalized with respect to values within the respective row by way of the softmax function. On the other hand, when the number of slot vectors corresponds to a number of rows of the attention matrix, each value within each respective column of the attention matrix may be normalized with respect to values within the respective column by way of the softmax function.

The slot attention module may also be configured to calculate an update matrix based on (i) the attention matrix and (ii) the plurality of feature vectors transformed by the value function. The neural network memory unit may be configured to update the plurality of slot vectors based on the update matrix and previous values of the slot vectors, thereby allowing the values of the slot vectors to be refined to improve their representational accuracy over one or more iterations of the slot attention module.

The slot vectors generated by the slot attention module may be permutation invariant with respect to the feature vectors and permutation equivariant with respect to one another. Thus, for a given initialization of the slot vectors, the order of the feature vectors of a given perceptual representation might not influence the values of the slot vectors and/or the order of the values of the slot vectors. Different initializations of the slot vectors may, for a given perceptual representation, change the order of the slot vectors, while the set of values of the slot vectors remains approximately constant. Thus, permuting the order of the slot vectors after initialization thereof may be equivalent to permuting the output of the slot attention module. The permutation equivariance of the slot vectors with respect to one another allows the slot vectors to be fully interchangeable and allows each slot vector to represent various entities independently of their types, classifications, and/or semantics.

The plurality of slot vectors may be used by one or more machine learning models to perform specific tasks, such as image reconstruction, text translation, object attribute/property detection, reward prediction, visual reasoning, question answering, control, and/or planning, among other possible tasks. Thus, the slot attention module may be trained jointly with the one or more machine learning models to generate slot vectors that are useful in carrying out the particular task of the one or more machine learning models. That is, the slot attention module may be trained to generate the slot vectors in a task-specific manner, such that the slot vectors represent the information important for the particular task and omit information that is not important and/or irrelevant for the particular task.

Although the slot attention module may be trained for a specific task, the architecture of the slot attention module is not task-specific and thus allows the slot attention module to be used for various tasks. The slot attention module may be used for both supervised and unsupervised training tasks. Additionally, the slot attention module does not assume, expect, or depend on the feature vectors representing a particular type of data (e.g., image data, waveform data, text data, etc). Thus, the slot attention module may be used with any type of data that can be represented by one or more feature vectors, and the type of data may be based on the task for which the slot attention module is used.

Further, the slot vectors themselves might not be specialized with respect to particular entity types and/or classifications. Thus, when multiple classes of entities are contained within the perceptual representation, each slot vector may be capable of representing each of the entities, regardless of its class. Each of the slot vectors may bind to or attach to a particular entity in order to represent its features, but this binding/attending is not dependent on entity type, classification, and/or semantics. The binding/attending of a slot vector to an entity may be driven by the downstream task for which the slot vectors are used—the slot attention module might not be “aware” of objects per-se, and might not distinguish between, for example, clustering objects, colors, and/or spatial regions.

II. Example Computing Devices

FIG. 1illustrates an example form factor of computing system100. Computing system100may be, for example, a mobile phone, a tablet computer, or a wearable computing device. However, other embodiments are possible. Computing system100may include various elements, such as body102, display106, and buttons108and110. Computing system100may further include front-facing camera104, rear-facing camera112, front-facing infrared camera114, and infrared pattern projector116.

Front-facing camera104may be positioned on a side of body102typically facing a user while in operation (e.g., on the same side as display106). Rear-facing camera112may be positioned on a side of body102opposite front-facing camera104. Referring to the cameras as front and rear facing is arbitrary, and computing system100may include multiple cameras positioned on various sides of body102. Front-facing camera104and rear-facing camera112may each be configured to capture images in the visible light spectrum.

Display106could represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some embodiments, display106may display a digital representation of the current image being captured by front-facing camera104, rear-facing camera112, and/or infrared camera114, and/or an image that could be captured or was recently captured by one or more of these cameras. Thus, display106may serve as a viewfinder for the cameras. Display106may also support touchscreen functions that may be able to adjust the settings and/or configuration of any aspect of computing system100.

Front-facing camera104may include an image sensor and associated optical elements such as lenses. Front-facing camera104may offer zoom capabilities or could have a fixed focal length. In other embodiments, interchangeable lenses could be used with front-facing camera104. Front-facing camera104may have a variable mechanical aperture and a mechanical and/or electronic shutter. Front-facing camera104also could be configured to capture still images, video images, or both. Further, front-facing camera104could represent a monoscopic, stereoscopic, or multiscopic camera. Rear-facing camera112and/or infrared camera114may be similarly or differently arranged. Additionally, one or more of front-facing camera104, rear-facing camera112, or infrared camera114, may be an array of one or more cameras.

Either or both of front-facing camera104and rear-facing camera112may include or be associated with an illumination component that provides a light field in the visible light spectrum to illuminate a target object. For instance, an illumination component could provide flash or constant illumination of the target object. An illumination component could also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields known and used to recover three-dimensional (3D) models from an object are possible within the context of the embodiments herein.

Infrared pattern projector116may be configured to project an infrared structured light pattern onto the target object. In one example, infrared projector116may be configured to project a dot pattern and/or a flood pattern. Thus, infrared projector116may be used in combination with infrared camera114to determine a plurality of depth values corresponding to different physical features of the target object.

Namely, infrared projector116may project a known and/or predetermined dot pattern onto the target object, and infrared camera114may capture an infrared image of the target object that includes the projected dot pattern. Computing system100may then determine a correspondence between a region in the captured infrared image and a particular part of the projected dot pattern. Given a position of infrared projector116, a position of infrared camera114, and the location of the region corresponding to the particular part of the projected dot pattern within the captured infrared image, computing system100may then use triangulation to estimate a depth to a surface of the target object. By repeating this for different regions corresponding to different parts of the projected dot pattern, computing system100may estimate the depth of various physical features or portions of the target object. In this way, computing system100may be used to generate a three-dimensional (3D) model of the target object.

Computing system100may also include an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene (e.g., in terms of visible and/or infrared light) that cameras104,112, and/or114can capture. In some implementations, the ambient light sensor can be used to adjust the display brightness of display106. Additionally, the ambient light sensor may be used to determine an exposure length of one or more of cameras104,112, or114, or to help in this determination.

Computing system100could be configured to use display106and front-facing camera104, rear-facing camera112, and/or front-facing infrared camera114to capture images of a target object. The captured images could be a plurality of still images or a video stream. The image capture could be triggered by activating button108, pressing a softkey on display106, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing button108, upon appropriate lighting conditions of the target object, upon moving digital camera device100a predetermined distance, or according to a predetermined capture schedule.

As noted above, the functions of computing system100may be integrated into a computing device, such as a wireless computing device, cell phone, tablet computer, laptop computer and so on. For purposes of example,FIG. 2is a simplified block diagram showing some of the components of an example computing device200that may include camera components224.

By way of example and without limitation, computing device200may be a cellular mobile telephone (e.g., a smartphone), a still camera, a video camera, a computer (such as a desktop, notebook, tablet, or handheld computer), personal digital assistant (PDA), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, or some other type of device. Computing device200may be equipped with at least some image capture and/or image processing capabilities, and/or audio capture and/or audio processing capabilities. It should be understood that computing device200may represent a physical image and/or audio processing system, a particular physical hardware platform on which an image and/or audio sensing and processing application operates in software, or other combinations of hardware and software that are configured to carry out image capture and/or processing functions and/or audio capture and/or processing functions.

As shown inFIG. 2, computing device200may include communication interface202, user interface204, processor206, data storage208, and camera components224, all of which may be communicatively linked together by a system bus, network, or other connection mechanism210.

Communication interface202may allow computing device200to communicate, using analog or digital modulation, with other devices, access networks, and/or transport networks. Thus, communication interface202may facilitate circuit-switched and/or packet-switched communication, such as plain old telephone service (POTS) communication and/or Internet protocol (IP) or other packetized communication. For instance, communication interface202may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface202may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port. Communication interface202may also take the form of or include a wireless interface, such as a Wi-Fi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPP Long-Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface202. Furthermore, communication interface202may comprise multiple physical communication interfaces (e.g., a Wi-Fi interface, a BLUETOOTH® interface, and a wide-area wireless interface).

User interface204may function to allow computing device200to interact with a human or non-human user, such as to receive input from a user and to provide output to the user. Thus, user interface204may include input components such as a keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone, and so on. User interface204may also include one or more output components such as a display screen which, for example, may be combined with a touch-sensitive panel. The display screen may be based on CRT, LCD, and/or LED technologies, or other technologies now known or later developed. User interface204may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and/or other similar devices. User interface204may also be configured to receive and/or capture audible utterance(s), noise(s), and/or signal(s) by way of a microphone and/or other similar devices.

In some embodiments, user interface204may include a display that serves as a viewfinder for still camera and/or video camera functions supported by computing device200(e.g., in both the visible and infrared spectrum). Additionally, user interface204may include one or more buttons, switches, knobs, and/or dials that facilitate the configuration and focusing of a camera function and the capturing of images. It may be possible that some or all of these buttons, switches, knobs, and/or dials are implemented by way of a touch-sensitive panel.

Processor206may comprise one or more general purpose processors—e.g., microprocessors—and/or one or more special purpose processors—e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, or application-specific integrated circuits (ASICs). In some instances, special purpose processors may be capable of image processing, image alignment, and merging images, among other possibilities. Data storage208may include one or more volatile and/or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with processor206. Data storage208may include removable and/or non-removable components.

Processor206may be capable of executing program instructions218(e.g., compiled or non-compiled program logic and/or machine code) stored in data storage208to carry out the various functions described herein. Therefore, data storage208may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by computing device200, cause computing device200to carry out any of the methods, processes, or operations disclosed in this specification and/or the accompanying drawings. The execution of program instructions218by processor206may result in processor206using data212.

By way of example, program instructions218may include an operating system222(e.g., an operating system kernel, device driver(s), and/or other modules) and one or more application programs220(e.g., camera functions, address book, email, web browsing, social networking, audio-to-text functions, text translation functions, and/or gaming applications) installed on computing device200. Similarly, data212may include operating system data216and application data214. Operating system data216may be accessible primarily to operating system222, and application data214may be accessible primarily to one or more of application programs220. Application data214may be arranged in a file system that is visible to or hidden from a user of computing device200.

Application programs220may communicate with operating system222through one or more application programming interfaces (APIs). These APIs may facilitate, for instance, application programs220reading and/or writing application data214, transmitting or receiving information via communication interface202, receiving and/or displaying information on user interface204, and so on.

In some vernaculars, application programs220may be referred to as “apps” for short. Additionally, application programs220may be downloadable to computing device200through one or more online application stores or application markets. However, application programs can also be installed on computing device200in other ways, such as via a web browser or through a physical interface (e.g., a USB port) on computing device200.

Camera components224may include, but are not limited to, an aperture, shutter, recording surface (e.g., photographic film and/or an image sensor), lens, shutter button, infrared projectors, and/or visible-light projectors. Camera components224may include components configured for capturing of images in the visible-light spectrum (e.g., electromagnetic radiation having a wavelength of 400-700 nanometers) and components configured for capturing of images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of 701 nanometers-1 millimeter). Camera components224may be controlled at least in part by software executed by processor206.

III. Example Slot Attention Module

FIG. 3illustrates a block diagram of slot attention module300. Slot attention module300may include value function308, key function310, query function312, slot attention calculator314, slot update calculator316, slot vector initializer318, and neural network memory unit320. Slot attention module300may be configured to receive as input perceptual representation302, which may include feature vectors304-306. Slot attention module300may be configured to generate slot vectors322-324based on perceptual representation302. Feature vectors304-306may represent a distributed representation of the entities in perceptual representation302, while slot vectors322-324may represent an entity-centric representation of these entities. Slot attention module300and the components thereof may represent a combination of hardware and/or software components configured to implement the functions described herein.

Perceptual representation302may represent various types of data, including, for example, two-dimensional image data (e.g., red-green-blue image data or grayscale image data), depth image data, point cloud data, audio data, time series data, or text data, among other possibilities. In some cases, perceptual representation302may be captured and/or generated by one or more sensors, such as visible light cameras (e.g., camera104), near-infrared cameras (e.g., infrared camera114), thermal cameras, stereoscopic cameras, time-of-flight (ToF) cameras, light detection and ranging (LIDAR) devices, radio detection and ranging (RADAR) devices, and/or microphones, among other possibilities. In other cases, perceptual representation302may additionally or alternatively include data generated by one or more users (e.g., words, sentences, paragraphs, and/or documents) or computing devices (e.g., rendered three-dimensional environments, time series plots), among other possibilities.

Perceptual representation302may be processed by way of one or more machine learning models to generate feature vectors304-306. Each feature vector of feature vectors304-306may include a plurality of values, with each value corresponding to a particular dimension of the feature vector. In some implementations, the plurality of values of each feature vector may collectively represent an embedding of at least a portion of perceptual representation302in a vector space defined by the one or more machine learning models. When perceptual representation302is an image, for example, each of feature vectors304-306may be associated with one or more pixels in the image, and may represent the various features of the one or more pixels. In some cases, the one or more machine learning models used to process perceptual representation302may include convolutional neural networks. Accordingly, feature vectors304-306may represent a map of convolutional features of perceptual representation302, and may thus include the outputs of various convolutional filters.

Each respective feature vector of feature vectors304-306may include a position embedding that indicates a portion of perceptual representation302represented by the respective feature vector. Feature vectors304-306may be determined, for example, by adding the position embedding to the convolutional features extracted from perceptual representation302. Encoding the position associated with each respective feature vector of feature vectors304-306as part of the respective feature vector, rather than by way of the order in which the respective feature vector is provided to slot attention module300, allows feature vectors304-306to be provided to slot attention module300in a plurality of different orders. Thus, including the position embeddings as part of feature vectors304-306enables slot vectors322-324generated by slot attention module300to be permutation invariant with respect to feature vectors304-306.

In the case of an image, for example, the position embedding may be generated by constructing a W×H×4 tensor, where W and H represent the width and height, respectively, of the map of the convolutional features of perceptual representation302. Each of the four values associated with each respective pixel along the W×H map may represent a position of the respective pixel relative to a border, boundary, and/or edge of the image along a corresponding direction (i.e., up, down, right, and left) of the image. In some cases, each of the four values may be normalized to a range from 0 to 1, inclusive. The W×H×4 tensor may be projected to the same dimension as the convolutional features (i.e., the same dimension as feature vectors304-306) by way of a learnable linear map. The projected W×H×4 tensor may then be added to the convolutional features to generate feature vectors304-306, thereby embedding feature vectors304-306with positional information. In some implementations, the sum of the projected W×H×4 tensor and the convolutional features may be processed by one or more machine learning models (e.g., one or more multi-layer perceptrons) to generate feature vectors304-306. Similar position embeddings may be included in feature vectors304-306for other types of perceptual representations as well.

Feature vectors304-306may be provided as input to key function310. Feature vectors304-306may include N vectors each having I dimensions. Thus, in some implementations, feature vectors304-306may be represented by an input matrix X having N rows (each corresponding to a particular feature vector) and I columns.

In some implementations, key function310may include a linear transformation represented by a key weight matrix WKEYhaving I rows and D columns. In other implementations, key function310may include a multi-layer perceptron that includes one or more hidden layers and that utilizes one or more non-linear activation functions. Key function310(e.g., key weight matrix WKEY) may be learned during training of slot attention module300. The input matrix X may be transformed by key function310to generate a key input matrix XKEY(e.g., XKEY=XWKEY), which may be provided as input to slot attention calculator314. Key input matrix XKEYmay include N rows and D columns.

Feature vectors304-306may also be provided as input to value function308. In some implementations, value function308may include a linear transformation represented by a value weight matrix WVALUEhaving I rows and D columns. In other implementations, value function308may include a multi-layer perceptron that includes one or more hidden layers and that utilizes one or more non-linear activation functions. Value function308(e.g., value weight matrix WVALUE) may be learned during training of slot attention module300. The input matrix X may be transformed by value function308to generate a value input matrix XVALUE(e.g., XVALUE=XWVALUE), which may be provided as input to slot update calculator316. Value input matrix XVALUEmay include N rows and D columns.

Since the dimensions of key weight matrix WKEYand the value weight matrix WVALUEdo not depend on the number N of feature vectors304-306, different values of N may be used during training and during testing/usage of slot attention module300. For example, slot attention module300may be trained on perceptual inputs with N=1024 feature vectors, but may be used with N=512 feature vectors or N=2048 feature vectors. However, since at least one dimension of the key weight matrix WKEYand the value weight matrix WVALUEdoes depend on the dimension I of feature vectors304-306, the same value of 1 may be used during training and during testing/usage of slot attention module300.

Slot vector initializer318may be configured to initialize each of slot vectors322-324stored by neural network memory unit320. In one example, slot vector initializer318may be configured to initialize each of slot vectors322-324with random values selected, for example, from a normal (i.e., Gaussian) distribution. In other examples, slot vector initializer318may be configured to initialize one or more respective slot vectors of slot vectors322-324with “seed” values configured to cause the one or more respective slot vectors to attend/bind to, and thereby represent, a particular entity contained within perceptual representation302. For example, when processing image frames of a video, slot vector initializer318may be configured to initialize slot vectors322-324for a second image frame based on the values of the slot vectors322-324determined with respect to a first image frame that precedes the second image frame. Accordingly, a particular slot vector of slot vectors322-324may be caused to represent the same entity across image frames of the video. Other types of sequential data may be similarly “seeded” by slot vector initializer318.

Slot vectors322-324may include K vectors each having S dimensions. Thus, in some implementations, slot vectors322-324may be represented by an output matrix Y having K rows (each corresponding to a particular slot vector) and S columns.

In some implementations, query function312may include a linear transformation represented by a query weight matrix WQUERYhaving S rows and D columns. In other implementations, query function312may include a multi-layer perceptron that includes one or more hidden layers and that utilizes one or more non-linear activation functions. Query function312(e.g., query weight matrix WQUERY) may be learned during training of slot attention module300. The output matrix Y may be transformed by query function312to generate a query input matrix YQUERY(e.g., YQUERY=YWQUERY), which may be provided as input to slot attention calculator314. Query output matrix YQUERYmay include K rows and D columns. Thus, the dimension D may be shared by value function308, key function310, and query function312.

Further, since the dimensions of the query weight matrix WQUERYdo not depend on the number K of slot vectors322-324, different values of K may be used during training and during testing/usage of slot attention module300. For example, slot attention module300may be trained with K=7 slot vectors, but may be used with K=5 slot vectors or K=11 slot vectors. Thus, slot attention module300may be configured to generalize across different numbers of slot vectors322-324without explicit training, although training and using slot attention module300with the same number of slot vectors322-324may improve performance. However, since at least one dimension of the query weight matrix WQUERYdoes depend on the dimension S of slot vectors322-324, the same value of S may be used during training and during testing/usage of slot attention module300.

Slot attention calculator314may be configured to determine attention matrix340based on key input matrix XKEYgenerated by key function310and query input matrix YQUERYgenerated by query function312. Specifically, slot attention calculator314may be configured to calculate a dot product between key input matrix XKEYand a transpose of query output matrix YQUERY. In some implementations, slot attention calculator314may also divide the dot product by the square root of D (i.e., the number of columns of the WVALUE, WKEY, and/or WQUERYmatrices). Thus, slot attention calculator314may implement the function M=(1/√{square root over (D)})XKEY(YQUERY)T, where M represents a non-normalized version of attention matrix340and may include N rows and K columns.

Slot attention calculator314may be configured to determine attention matrix340by normalizing the values of the matrix M with respect to the output axis (i.e., with respect to slot vectors322-324). Thus, the values of the matrix M may be normalized along the rows thereof (i.e., along the dimension K corresponding to the number of slot vectors322-324). Accordingly, each value in each respective row may be normalized with respect to the K values contained in the respective row.

Thus, slot attention calculator314may be configured to determine attention matrix340by normalizing each respective value of a plurality of values of each respective row of the matrix M with respect to the plurality of values of the respective row. Specifically, slot attention calculator314may determine attention matrix340according to Ai,j=(eMi,j)/(Σl=1KeMi,l), where Aijindicates the value at a position corresponding to row i and column j of attention matrix340, which may be alternatively referred to as attention matrix A. Normalizing the matrix M in this manner may cause slots to compete with one another for representing a particular entity. The function implemented by slot attention calculator314for computing Ai,jmay be referred to as a softmax function. Attention matrix A (i.e., attention matrix340) may include N rows and K columns.

In other implementations, the matrix M may be transposed prior to normalization, and the values of the matrix MTmay thus be normalized along the columns thereof (i.e., along the dimension K corresponding to the number of slot vectors322-324). Accordingly, each value in each respective column of the matrix MTmay be normalized with respect to the K values contained in the respective column. Slot attention calculator314may determine a transposed version of attention matrix340according to Ai,jT=(eMi,j)/(Σl=1KeMi,j), where Ai,jTindicates the value at a position corresponding to row i and column j of transposed attention matrix340, which may be alternatively referred to as transposed attention matrix AT. Nevertheless, transposed attention matrix340may still be determined by normalizing the values of the matrix M with respect to the output axis (i.e., with respect to slot vectors322-324).

Slot update calculator316may be configured to determine update matrix342based on value input matrix XVALUEgenerated by value function308and attention matrix340. In one implementation, slot update calculator316may be configured to determine update matrix342by determining a dot product of a transpose of the attention matrix A and the value input matrix XVALUE. Thus, slot update calculator316may implement the function UWEIGHTED SUM=ATXVALUE, where the attention matrix A may be viewed as specifying the weights of a weighted sum calculation and the value input matrix XVALUEmay be viewed as specifying the values of the weighted sum calculation. Update matrix342may thus be represented by UWEIGHTED SUM, which may include K rows and D columns.

In another implementation, slot update calculator316may be configured to determine update matrix342by determining a dot product of a transpose of an attention weight matrix WATTENTIONand the value input matrix XVALUE. Elements/entries of the attention weight matrix WATTENTIONmay be defined as Wi,jATTENTION=(Ai,j)/(Σl=1NAl,j), or, for the transpose thereof, as (Wi,jATTENTION)T=(Ai,jT)/(Σl=1NAi,lT). Thus, slot update calculator316may implement the function UWEIGHTED MEAN=(WATTENTION)TXVALUE, where the matrix A may be viewed as specifying the weights of a weighted mean calculation and the value input matrix XVALUEmay be viewed as specifying the values of the weighted mean calculation. Update matrix342may thus be represented by UWEIGHTED MEAN, which may include K rows and D columns.

Update matrix342may be provided as input to neural network memory unit320, which may be configured to update slot vectors322-324based on the previous values of slot vectors322-324and update matrix342. Neural network memory unit320may include a gated recurrent unit (GRU) and/or a long-short term memory (LSTM) network, as well as other neural network or machine learning-based memory units configured to store and/or update slot vectors322-324. For example, in addition to a GRU and/or an LSTM, neural network memory unit320may include one or more feed-forward neural network layers configured to further modify the values of slot vectors322-324after modification by the GRU and/or LSTM (and prior to being provided to task-specific machine learning model330).

In some implementations, neural network memory unit320may be configured to update each of slot vectors322-324during each processing iteration, rather than updating only some of slot vectors322-324during each processing iteration. Training neural network memory unit320to update the values of slot vectors322-324based on the previous values thereof and based on update matrix342, rather than using update matrix342as the updated values of slot vectors322-324, may improve the accuracy and/or speed up convergence of slot vectors322-324.

Slot attention module300may be configured to generate slot vectors322-324in an iterative manner. That is, slot vectors322-324may be updated one or more times before being passed on as input to task-specific machine learning model330. For example, slot vectors322-324may be updated three times before being considered “ready” to be used by task-specific machine learning model330. Specifically, the initial values of slot vectors322-324may be assigned thereto by slot vector initializer318. When the initial values are random, they likely will not accurately represent the entities contained in perceptual representation302. Thus, feature vectors304-306and the randomly-initialized slot vectors322-324may be processed by components of slot attention module300to refine the values of slot vectors322-324, thereby generating updated slot vectors322-324.

After this first iteration or pass through slot attention module300, each of slot vectors322-324may begin to attend to and/or bind to, and thus represent, one or more corresponding entities contained in perceptual representation302. Feature vectors304-306and the now-updated slot vectors322-324may again be processed by components of slot attention module300to further refine the values of slot vectors322-324, thereby generating another update to slot vectors322-324. After this second iteration or pass through slot attention module300, each of slot vectors322-324may continue to attend to and/or bind to the one or more corresponding entities with increasing strength, thereby representing the one or more corresponding entities with increasing accuracy.

Further iterations may be performed, and each additional iteration may generate some improvement to the accuracy with which each of slot vectors322-324represents its corresponding one or more entities. After a predetermined number of iterations, slot vectors322-324may converge to an approximately stable set of values, resulting in no additional accuracy improvements. Thus, the number of iterations of slot attention module300may be selected based on (i) a desired level of representational accuracy for slot vectors322-324and (ii) desired processing time before slot vectors322-324are usable by task-specific machine learning model330.

Task-specific machine learning model330may represent a plurality of different tasks, including both supervised and unsupervised learning tasks. Example implementations of task-specific machine learning model330are illustrated in and discussed with respect toFIGS. 5A and 5B. Task-specific machine learning model330may be co-trained with slot attention module300. Thus, depending on the specific task associated with task-specific machine learning model330, slot attention module300may be trained to generate slot vectors322-324that are adapted for and provide values useful in executing the specific task. Specifically, learned parameters associated with one or more of value function308, key function310, query function312, and/or neural network memory unit320may vary as a result of training based on the specific task associated with task-specific machine learning model330. In some implementations, slot attention module300may be trained using adversarial training and/or contrastive learning, among other training techniques.

Slot attention module300may take less time to train (e.g., 24 hours, compared to 7 days for an alternative approach executed on the same computing hardware) and consume fewer memory resources (e.g., allowing for a batch size of 64, compared to a batch size of 4 for the alternative approach executed on the same computing hardware) than alternative approaches for determining entity-centric representations. In some implementations, slot attention module300may also include one or more layer normalizations. For example, layer normalizations may be applied to feature vectors304-306prior to the transformation thereof by the key function310, to slot vectors322-324prior to transformation thereof by query function312, and/or to slot vectors322-324after being at least partially updated by neural network memory unit320. Layer normalizations may improve the stability and speed up the convergence of slot attention module300.

IV. Example Slot Vectors

FIG. 4graphically illustrates an example of a plurality of slot vectors changing over the course of processing iterations by slot attention module300with respect to a particular perceptual representation. In this example, perceptual representation302is represented by image400that includes three entities: entity410(i.e., a circular object); entity412(i.e., a square object); and entity414(i.e., a triangular object). Image400may be processed by one or more machine learning models to generate feature vectors304-306, each represented by a corresponding grid element of the grid overlaid on top of image400. Thus, a leftmost grid element in the top row of the grid may represent feature vector304, a rightmost grid element in the bottom row of the grid may represent feature vector306, and grid elements therebetween may represent other feature vectors. Thus, each grid element may represent a plurality of vector values associated with the corresponding feature vector.

FIG. 4illustrates the plurality of slot vectors as having four slot vectors. However, in general, the number of slot vectors may be modifiable. For example, the number of slot vectors may be selected to be at least equal to a number of entities expected to be present in perceptual representation302so that each entity may be represented by a corresponding slot vector. Thus, in the example illustrated inFIG. 4, the four slot vectors provided exceed the number of entities (i.e., the three entities410,412, and414) contained in image400. In cases where the number of entities exceeds the number of slot vectors, one or more slot vectors may represent two or more entities.

Slot attention module300may be configured to process the feature vectors associated with image400and the initial values of the four slot vectors (e.g., randomly initialized) to generate slot vectors with values402A,404A,406A, and408A. Slot vector values402A,404A,406A, and408A may represent the output of a first iteration (1×) of slot attention module300. Slot attention module300may also be configured to process the feature vectors and slot vectors with values402A,404A,406A, and408A to generate slot vectors with values402B,404B,406B, and408B. Slot vector values402B,404B,406B, and408B may represent the output of a second iteration (2×) of slot attention module300. Slot attention module300may be further configured to process the feature vectors and slot vectors with values402B,404B,406B, and408B to generate slot vectors with values402C,404C,406C, and408C. Slot vector values402C,404C,406C, and408C may represent the output of a third iteration (3×) of slot attention module300. The visualizations of slot vector values402A,404A,406A,408A,402B,404B,406B,408B,402C,404C,406C,408C may represent visualizations of attention masks based on attention matrix340at each iteration and/or visualizations of reconstruction masks generated by task-specific machine learning model330, among other possibilities.

The first slot vector (associated with values402A,402B, and402C) may be configured to attend to and/or bind to entity410, thereby representing attributes, properties, and/or characteristics of entity410. Specifically, after the first iteration of slot attention module300, the first slot vector may represent aspects of entity410and entity412, as shown by the black-filled regions in the visualization of slot vector values402A. After the second iteration of slot attention module300, the first slot vector may represent a larger portion of entity410and a smaller portion of entity412, as shown by the increased black-filled region of entity410and decreased black-filled region of entity412in the visualization of slot vector values402B. After the third iteration of slot attention module300, the first slot vector may represent entity410approximately exclusively, and might no longer represent entity412, as shown by entity410being completely black-filled and entity412being illustrate completely white-filled in the visualization of slot vector values402C. Thus, the first slot vector may converge and/or focus on representing entity410as slot attention module300updates and/or refines the values of the first slot vector. This attention and/or convergence of a slot vector to one or more entities is a result of the mathematical structure of components of slot attention module300and task-specific training of slot attention module300.

The second slot vector (associated with values404A,404B, and404C) may be configured to attend to and/or bind to entity412, thereby representing attributes, properties, and/or characteristics of entity412. Specifically, after the first iteration of slot attention module300, the second slot vector may represent aspects of entity412and entity410, as shown by the black-filled regions in the visualization of slot vector values404A. After the second iteration of slot attention module300, the second slot vector may represent a larger portion of entity412and might no longer represent entity410, as shown by the increased black-filled region of entity412and entity410being illustrated completely white-filled in the visualization of slot vector values404B. After the third iteration of slot attention module300, the second slot vector may represent entity412approximately exclusively, and might continue to no longer represent entity410, as shown by entity412being completely black-filled and entity410being completely white-filled in the visualization of slot vector values404C. Thus, the second slot vector may converge and/or focus on representing entity412as slot attention module updates and/or refines the values of the second slot vector.

The third slot vector (associated with values406A,406B, and406C) may be configured to attend to and/or bind to entity414, thereby representing attributes, properties, and/or characteristics of entity414. Specifically, after the first iteration of slot attention module300, the third slot vector may represent aspects of entity414, as shown by the black-filled regions in the visualization of slot vector values406A. After the second iteration of slot attention module300, the third slot vector may represent a larger portion of entity414, as shown by the increased black-filled region of entity414in the visualization of slot vector values404B. After the third iteration of slot attention module300, the third slot vector may represent approximately the entirety of entity414, as shown by entity412being completely black-filled in the visualization of slot vector values406C. Thus, the third slot vector may converge and/or focus on representing entity414as slot attention module updates and/or refines the values of the third slot vector.

The fourth slot vector (associated with values408A,408B, and408C) may be configured to attend to and/or bind to the background features of image400, thereby representing attributes, properties, and/or characteristics of the background. Specifically, after the first iteration of slot attention module300, the fourth slot vector may represent approximately the entirety of the background and respective portions of entities410and414that are not already represented by slot vector values402A404A, and/or406A, as shown by the black-filled region in the visualization of slot vector values408A. After the second iteration of slot attention module300, the fourth slot vector may represent approximately the entirety of the background and smaller portions of entities410and414not already represented by slot vector values402B404B, and/or406B, as shown by the black-filled region of the background and decreased black-filled region of entities410and414in the visualization of slot vector values408B. After the third iteration of slot attention module300, the fourth slot vector may approximately exclusively represent approximately the entirety of the background, as shown by the background being completely black-filled and entities410,412, and414being completely white-filled in the visualization of slot vector values408C. Thus, the fourth slot vector may converge and/or focus on representing the background of image400as slot attention module updates and/or refines the values of the fourth slot vector.

In some implementations, rather than representing the background of image400, the fourth slot vector may instead take on a predetermined value indicating that the fourth slot vector is not utilized to represent an entity. Thus, the background may be unrepresented. Alternatively or additionally, when additional slot vectors are provided (e.g., a fifth slot vector), the additional vectors may represent portions of the background or may be unutilized. Thus, in some cases, slot attention module300may distribute the representation of the background among multiple slot vectors. In some implementations, the slot vectors might treat the entities within the perceptual representation the same as the background thereof. Specifically, any one of the slot vectors may be used to represent the background and/or an entity (e.g., the background may be treated as another entity). Alternatively, in other implementations, one or more of the slot vectors may be reserved to represent the background.

The plurality of slot vectors may be invariant with respect to an order of the feature vectors and equivariant with respect to one another. That is, for a given initialization of the slot vectors, the order in which the feature vectors are provided at the input to slot attention module300does not affect the order and/or values of the slot vectors. However, different initializations of the slot vectors may affect the order of the slot vectors regardless of the order of the feature vectors. Further, for a given set of feature vectors, the set of values of the slot vectors may remain constant, but the order of the slot vectors may be different. Thus, different initializations of the slot vectors may affect the pairings between slot vectors and entities contained in the perceptual representation, but the entities may nevertheless be represented with approximately the same set of slot vector values.

V. Example Slot Attention Module Applications

FIG. 5Aillustrates an example application of slot attention module300to an unsupervised learning task. Specifically, convolutional neural network models502may be used to generate perceptual representation302based on input data500, which may represent image data, time series (e.g., waveform) data, text data, point cloud data, and/or voxel data, among other types of input. Perceptual representation302may include feature vectors304-306, which may represent results of processing of input data500by convolutional neural network models502and/or aspects of unprocessed input data500. Perceptual representation302may be provided as input to slot attention module300, which may be configured to generate slot vectors322-324based thereon.

Slot decoder model506may be configured to receive slot vectors322-324as input and, based thereon, generate input data reconstruction508. The values of slot vectors322-324provided to slot decoder model506may represent the output of one or more iterations of processing by slot attention module300. Slot attention module300and slot decoder model506may be trained jointly, thereby resulting in slot vectors322-324providing embeddings of entities present in input data500that can be used by slot decoder model506to reconstruct input data500(i.e., generate input data reconstruction508). That is, co-training of slot attention module300and slot decoder model506allows slot decoder model506to “understand” the values of slot vectors322-324.

In one example, input data500may represent image data, and slot decoder model506may individually decode each of slot vectors322-324using a spatial broadcast decoder. Specifically, each slot may be broadcast onto a two-dimensional grid which may be augmented with position embeddings. Each grid may be decoded using a convolutional neural network (the parameters of which may be shared across each of slot vectors322-324) to generate an output of size W×H×4, where W and H represent the width and height, respectively, of the reconstructed slot-specific image data and the additional 4 dimensions represent the red, green, and blue color channels and a non-normalized alpha mask thereof. The alpha masks may be normalized across the slot-specific images using a softmax function and may be used as mixture weights to recombine and/or mix the slot-specific images into a final reconstruction of the original image data (i.e., input data reconstruction508). In other examples, slot decoder model506may be and/or may include aspects of patch-based decoders.

Although slot attention module300may be used in combination with slot decoder model506, the architecture of slot attention module300does not itself include or depend on any decoders and/or decoding operations. The functionality provided by a decoder in other attention-based neural network architectures is instead replaced by the iterative processing carried out by slot attention module—i.e., processing slot vectors322-324multiple times to achieve an accurate representation of entities in perceptual representation302. Thus, slot attention module300may be used in applications beyond autoencoding, such as contrastive representation learning for object discovery and/or direct optimization of a downstream task, such as control planning.

Slot vectors322-324may collectively define latent representation504of input data500. In some cases, latent representation504may represent a compression of the information contained in input data500. Thus, in some implementations, slot attention module300may be used as and or viewed as a machine learning encoder. Accordingly, slot attention module300may be used for image reconstruction, text translation, and/or other applications that utilize machine learning encoders. Unlike certain other latent representations, each slot vector of latent representation504may capture the properties of corresponding one or more entities in perceptual representation302, and may do so without relying on assumption about an order in which the entities are described by perceptual representation302.

Further, reconstruction of input data500(i.e., generation of input data reconstruction508) may be viewed as an unsupervised learning task at least because the training process may be carried out without assigning any explicit labels to input data500. The accuracy of input data reconstruction508may be determined by comparing input data reconstruction508to input data500, rather than to labels assigned to features in input data500.

FIG. 5Billustrates an example application of slot attention module300to a supervised learning task. Specifically, convolutional neural network models502and slot attention module300may operate on input data500and perceptual representation302, respectively, as discussed with respect toFIG. 5A, thereby generating slot vectors322-324. Entity attribute model510may be configured to determine, based on slot vectors322-324, a plurality of attributes of one or more entities represented in input data500.

Specifically, entity attribute model510may be configured to generate attributes for entities512-520. Thus, entity attribute model510may be configured to determine attributes514-516for entity512and entity attributes522-524for entity520. When input data500represents an image, for example, entities512-520may represent various objects or environmental features depicted by the image, and attributes514-516and522-524may represent various properties of the corresponding objects and/or features. For example, attributes514and522may indicate a color of entities512and520, respectively, while attributes516and524may indicate a shape of entities512and520, respectively.

Entity attribute model510may represent a machine learning model, such as an artificial neural network having a plurality of layers. Entity attribute model510may be trained jointly with slot attention module300, thereby resulting in slot vectors322-324providing embeddings of entities512-520present in input data500that can be used by entity attribute model510to determine the attributes of entities512-520. That is, co-training of slot attention module300and entity attribute model510allows entity attribute model510to “understand” the values of slot vectors322-324. Further, determination of the attributes of entities512-520may be viewed as a supervised learning task at least because the training process may utilize labels assigned to input data500to indicate the ground-truth values of the attributes of entities512-520. The accuracy of the attributes of entities512-520determined by entity attribute model510may be determined by comparing these attributes to the ground-truth values.

Additionally, since slot attention module300is permutation equivariant with respect to slot vectors322-324, training of this system may involve using a matching algorithm to match the individual outputs of entity attribute model510to corresponding ground-truth values. That is, different training iterations may result in a reordering of the pairing between slot vectors322-324and corresponding entities512-520. As such reordering takes place, each of slot vectors322-324and corresponding entities512-520may be re-paired with the set of ground-truth data corresponding thereto using, for example, the Hungarian algorithm.

VI. Example Operations

FIG. 6illustrates a flow chart of operations related to determining entity-centric representations of one or more entities contained in a perceptual representation. The operations may be carried out by computing system100, computing device200, and/or slot attention module300, among other possible types of devices or device subsystems. The embodiments ofFIG. 6may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and/or implementations of any of the previous figures or otherwise described herein.

Block600may involve receiving a perceptual representation comprising a plurality of feature vectors.

Block602may involve initializing a plurality of slot vectors represented by a neural network memory unit. Each respective slot vector of the plurality of slot vectors may be configured to represent a corresponding entity contained in the perceptual representation.

Block604may involve determining an attention matrix based on a product of (i) the plurality of feature vectors transformed by a key function and (ii) the plurality of slot vectors transformed by a query function. Each respective value of a plurality of values along each respective dimension of a plurality of dimensions of the attention matrix may be normalized with respect to the plurality of values along the respective dimension.

Block606may involve determining an update matrix based on (i) the plurality of feature vectors transformed by a value function and (ii) the attention matrix.

Block608may involve updating the plurality of slot vectors based on the update matrix by way of the neural network memory unit.

In some embodiments, a second attention matrix may be determined based on a product of (i) the plurality of feature vectors transformed by the key function and (ii) the plurality of updated slot vectors transformed by the query function. Each respective value of a plurality of values along each respective dimension of a plurality of dimensions of the second attention matrix may be normalized with respect to the plurality of values along the respective dimension of the second attention matrix. A second update matrix may be determined based on (i) the plurality of feature vectors transformed by the value function and (ii) the second attention matrix. The plurality of updated slot vectors may be further updated based on the second update matrix by way of the neural network memory unit.

In some embodiments, each respective slot vector may represent a semantic embedding of the corresponding entity.

In some embodiments, updating the respective slot vector may iteratively refine the representation of the corresponding entity and may bind the respective slot vector to the corresponding entity.

In some embodiments, the plurality of dimensions of the attention matrix may include a plurality of rows of the attention matrix. Each respective value of the plurality of values of each respective row of the plurality of rows of the attention matrix may be normalized with respect to the plurality of values of the respective row by way of a softmax function by dividing (i) an exponent of the respective value of the plurality of values of the respective row by (ii) a sum of exponents of the plurality of values of the respective row.

In some embodiments, normalizing each respective value of the plurality of values of each respective row of the plurality of rows of the attention matrix with respect to the plurality of values of the respective row by way of the softmax function may cause the plurality of slot vectors to compete with one another for representing entities contained in the perceptual representation.

In some embodiments, determining the update matrix may include determining a product of (i) the plurality of feature vectors transformed by the value function and (ii) a transpose of the attention matrix.

In some embodiments, determining the update matrix may include determining an attention weight matrix by dividing (i) each respective value of a plurality of values of each respective column of a plurality of columns of the attention matrix by (ii) a sum of the plurality of values in the respective column, and determining a product of (i) the plurality of feature vectors transformed by the value function and (ii) a transpose of the attention weight matrix.

In some embodiments, the plurality of slot vectors may include K slot vectors. Each respective dimension of the plurality of dimensions (e.g., each respective row of the plurality of rows of the attention matrix) may include K values. Each respective value of the K values along each respective dimension of the plurality of dimensions of the attention matrix may be normalized with respect to the K values along the respective dimension by way of a softmax function by dividing (i) an exponent of the respective value of the K values along the respective dimension by (ii) a sum of exponents of the K values along the respective dimension.

In some embodiments, normalizing each respective value of the K values along each respective dimension of the plurality of dimensions of the attention matrix with respect to the K values of the respective dimension by way of the softmax function may cause the plurality of slot vectors to compete with one another for representing entities contained in the perceptual representation.

In some embodiments, the plurality of feature vectors may include N feature vectors. The plurality of dimensions may be a first plurality of dimensions comprising N dimensions (e.g., N rows). Determining the update matrix may include: determining an attention weight matrix by dividing (i) each respective value of N values along each respective dimension of a second plurality of dimensions (e.g., columns) of the attention matrix by (ii) a sum of the N values along the respective dimension of the second plurality of dimensions, and determining a product of (i) the plurality of feature vectors transformed by the value function and (ii) a transpose of the attention weight matrix.

In some embodiments, one or more of: (i) the key function, (ii) the query function, (iii) the value function, or (iv) parameters of the neural network memory unit may be learned during training.

In some embodiments, the plurality of feature vectors may be represented by an input matrix that includes (i) N rows corresponding to a number of the plurality of feature vectors and (ii) I columns corresponding to a number of dimensions of each of the plurality of feature vectors. The key function may include a linear transformation represented by a key weight matrix comprising I rows and D columns.

In some embodiments, determining the attention matrix based on the product may include determining a dot product of (i) the plurality of feature vectors transformed by the key function and (ii) a transpose of the plurality of slot vectors transformed by the query function, and dividing the dot product by a square root of D.

In some embodiments, the plurality of slot vectors may be represented by a slot matrix that includes (i) K rows corresponding to a number of the plurality of slot vectors and (ii) S columns corresponding to a number of dimensions of each of the plurality of slot vectors. The query function may include a linear transformation represented by a query weight matrix comprising S rows and D columns.

In some embodiments, the plurality of feature vectors may be represented by an input matrix that includes (i) N rows corresponding to a number of the plurality of feature vectors and (ii) I columns corresponding to a number of dimensions of each of the plurality of feature vectors. The value function may include a linear transformation represented by a value weight matrix comprising I rows and D columns.

In some embodiments, the plurality of slot vectors may be permutation equivariant with respect to one another such that, for multiple different initializations of the plurality of slot vectors with respect to a given perceptual representation, a set of values of the plurality of slot vectors is approximately constant and an order of the plurality of slot vectors is variable.

In some embodiments, the plurality of slot vectors may be permutation invariant with respect to the plurality of feature vectors such that, for multiple different permutations of the plurality of feature vector, a set of values of the plurality of slot vectors is approximately constant.

In some embodiments, the respective slot vector may be configured to specialize in representing the corresponding entity contained in the perceptual representation independently of a classification of the corresponding entity.

In some embodiments, the perceptual representation may include one or more of: (i) two-dimensional image data, (ii) depth image data, (iii) point cloud data, (iv) time series data, (v) audio data, or (vi) text data. The perceptual representation may be processed by way of one or more machine learning models to generate the plurality of feature vectors.

In some embodiments, the one or more machine learning models may include a convolutional neural network.

In some embodiments, each respective feature vector of the plurality of feature vectors may include a position embedding that indicates a portion of the perceptual representation represented by the respective feature vector.

In some embodiments, the corresponding entity represented by the respective slot vector may include one or more of: (i) an object, (ii) a surface, (iii) a background, (iv) a waveform pattern, or (v) one or more words.

In some embodiments, the neural network memory unit may include at least one of: (i) a gated recurrent unit (GRU) or (ii) a long-short term memory neural network (LSTM).

In some embodiments, updating the plurality of slot vectors based on the update matrix by way of the neural network memory unit may include processing the update matrix by way of the neural network memory unit and updating the plurality of slot vectors by way of a feed-forward artificial neural network connected to an output of the neural network memory unit.

In some embodiments, a supervised learning task may be performed based on the updated plurality of slot vectors.

In some embodiments, the supervised learning task may be performed by one or more machine learning models that are jointly trained with one or more of the key function, the query function, the value function, or the neural network memory unit to perform the supervised learning task.

In some embodiments, the supervised learning task may include determining, by way of one or more machine learning models and based on the plurality of slot vectors, one or more attributes of entities represented by the perceptual representation.

In some embodiments, an unsupervised learning task may be performed based on the updated plurality of slot vectors.

In some embodiments, the unsupervised learning task may be performed by one or more machine learning models that are jointly trained with one or more of the key function, the query function, the value function, or the neural network memory unit to perform the unsupervised learning task.

In some embodiments, the unsupervised learning task may include determining, by way of one or more machine learning models and based on the plurality of slot vectors, a reconstruction of the perceptual representation.

In some embodiments, a layer normalization may be applied to one or more of: (i) the plurality of feature vectors prior to the transformation thereof by the key function, (ii) the plurality of slot vectors prior to transformation thereof by the query function, or (iii) the updated plurality of slot vectors after updating the plurality of slot vectors based on the update matrix.

In some embodiments, initializing the plurality of slot vectors may include initializing the plurality of slot vectors with values selected from a normal distribution.

In some embodiments, initializing the plurality of slot vectors may include initializing the plurality of slot vectors based on values of one or more preceding slot vectors determined for a preceding perceptual representation processed before the perceptual representation. Initializing the plurality of slot vectors based on the values of the one or more preceding slot vectors may cause the plurality of slot vectors to track entities across successive perceptual representations.

In some embodiments, the perceptual representation may include a first image frame of a video. Initializing the plurality of slot vectors may include initializing the plurality of slot vectors with values determined for the plurality of slot vectors based on a second image frame preceding the first image frame within the video to cause the plurality of slot vectors to track entities across image frames of the video.

In some embodiments, a number of slot vectors in the plurality of slot vectors may be adjustable.

In some embodiments, when a number of slot vectors in the plurality of slot vectors exceeds a number of entities contained in the perceptual representation, values of one or more slot vector of the plurality of slot vectors may be configured to indicate that the one or more slot vectors are unused. When the number of entities contained in the perceptual representation exceeds the number of slot vectors in the plurality of slot vectors, at least one slot vector of the plurality of slot vectors may be configured to represent multiple corresponding entities contained in the perceptual representation.

VII. Example Testing Results

Table 1 illustrates Adjusted Rand Index scores (in %, mean+/−standard deviation) comparing the performance of slot attention module300to other machine learning architectures, including IODINE, MONet, and Slot MLP, in an unsupervised object discovery task. Slot MLP represents a multi-layer perceptron model that, for performance testing purposes, replaces slot attention module300and instead maps the feature vectors to the slot vectors. CLEVR6, Multi-dSprites, and Tetrominoes represent multi-object data sets used as benchmarks in evaluating image processing models. As can be seen from Table 1, slot attention module300performs at least as well as the other machine learning architectures and, in most cases, outperforms the other machine learning architectures. Slot attention module300may similarly match or outperform the other machine learning architectures in performing tasks other than unsupervised object discovery.

The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example embodiments described herein and in the figures are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

The computer readable medium may also include non-transitory computer readable media such as computer readable media that store data for short periods of time like register memory, processor cache, and RAM. The computer readable media may also include non-transitory computer readable media that store program code and/or data for longer periods of time. Thus, the computer readable media may include secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, solid state drives, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.