MULTI-RESOLUTION NEURAL NETWORK ARCHITECTURE SEARCH SPACE FOR DENSE PREDICTION TASKS

Systems and methods for searching a search space are disclosed. Some examples may include using a first parallel module including a first plurality of stacked searching blocks and a second plurality of stacked searching blocks to output first feature maps of a first resolution and to output second feature maps of a second resolution. In some examples, a fusion module may include a plurality of searching blocks, where the fusion module is configured to generate multiscale feature maps by fusing one or more feature maps of the first resolution received from the first parallel module with one or more feature maps of the second resolution received from the first parallel module, and wherein the fusion module is configured to output the multiscale feature maps and output third feature maps of a third resolution.

BACKGROUND

Neural architecture search (NAS) is a technique used to automate the design of artificial neural networks (ANN), a frequently used model in the area of machine learning. NAS has been used to design networks that can outperform hand-designed architectures. Methods for NAS can be categorized according to the search space, search strategy, and performance estimation strategy used, where the search space defines the type(s) of ANN that can be designed and optimized, the search strategy defines the process used to explore the search space, and the performance estimation strategy evaluates the performance of an ANN based on its design.

In image and computer vision tasks, high-resolution representations (HR) are essential for dense prediction tasks such as segmentation, detection, and pose estimation. Learning HR representations is typically ignored in previous NAS methods that focus on image classification. While NAS methods have achieved success in automatically designing efficient models for image classification and to improve efficiency of models for dense prediction tasks such as semantic segmentation and pose estimation, existing NAS methods for dense prediction either directly extend the search space designed for image classification or only search for a feature aggregation head. This lack of consideration to the specificity of dense prediction hinders the performance advancement of NAS methods compared to the best hand-crafted models.

In principle, dense prediction tasks require integrity of global context and high-resolution representations. The former is critical to clarify ambiguous local features at each pixel, and the latter is useful for accurate predictions of fine details, such as semantic boundaries and key point locations. However, the integrity of global context and high-resolution representations have not been the focus of prominent NAS algorithms for classification. Commonly, multi-scale features have been combined at the end of the network while recent approaches have increased performance by putting multi-scale feature processing within the network backbone. In addition, multi-scale convolutional representations do not provide a global outlook of the image since dense prediction tasks often come with high input resolution, while a network often covers a fixed receptive field. Therefore, global attention strategies such as Squeeze-and-Excitation Network (SENet) or non-local networks have been proposed to enrich image convolutional features. Transformers, widely used in natural language processing, have exhibited good results when combined with a convolutional neural network for image classification and object detection. However, the computational complexity associated with transformers increases quadratically with the number of pixels; thus, a transformer implementation has been known to be computationally expensive.

It is with respect to these and other general considerations that embodiments have been described. Although relatively specific problems have been discussed, it should be understood that the examples described herein should not be limited to solving the specific problems identified in the background above.

SUMMARY

In accordance with examples of the present disclosure, systems and methods directed to high-resolution Neural Architecture Search (HR-NAS) are described. HR-NAS implementations described herein can find efficient and accurate networks for different tasks, by effectively encoding multiscale contextual information while maintaining high-resolution representations. To better encode multiscale image contexts in the search space of HR-NAS, a lightweight transformer having a computational complexity that can be dynamically changed with respect to different objective functions and computation budgets is utilized. In order to maintain high-resolution representations of learned networks, HR-NAS makes use of a multi-branch architecture that provides convolutional encoding of multiple feature resolutions. Accordingly, an efficient fine-grained search strategy can be used to train HR-NAS, which effectively explores the search space and determines optimal architectures given various tasks and computation resources.

In accordance with at least one example of the present disclosure, a search space is described. The search space may include a first parallel module including a first plurality of stacked searching blocks and a second plurality of stacked searching blocks, wherein the first plurality of stacked searching blocks is configured to output first feature maps of a first resolution and the second plurality of stacked searching blocks is configured to output second feature maps of a second resolution; a fusion module including a plurality of searching blocks, wherein the fusion module is configured to generate multiscale feature maps by fusing one or more feature maps of the first resolution received from the first parallel module with one or more feature maps of the second resolution received from the first parallel module, and wherein the fusion module is configured to output the multiscale feature maps and output third feature maps of a third resolution; and a second parallel module configured to receive the multiscale feature maps and the third feature maps of the third resolution from the fusion module, and output fourth feature maps of the first resolution, fifth feature maps of the second resolution, and sixth feature maps of the third resolution.

In accordance with examples of the present disclosure, a search space is described. The search space may include a first branch including a first plurality of stacked searching blocks for image features of a first resolution, one or more searching blocks of the first plurality of stacked searching blocks including a plurality of convolution layers and at least one transformer configured to provide an attention map based on image features from another searching block of the first branch; a second branch including a second plurality of stacked searching blocks for image features of a second resolution, one or more searching blocks of the second plurality of stacked searching blocks including a plurality of convolution layers and at least one transformer configured to provide an attention map based on image features from another searching block of the second branch; and a fusion module configured to fuse image features output by the one or more searching blocks of the first plurality of stacked searching blocks and image features output by the one or more searching blocks of the second plurality of stacked searching blocks, wherein the fusion module is configured to output image features of the first resolution and image features of the second resolution.

In accordance with examples of the present disclosure, a method of searching a search space is described. The method may include generating image features of a first resolution using a first parallel module including a first plurality of stacked searching blocks, wherein one or more searching blocks of the first plurality of stacked searching blocks includes a plurality of convolution layers and at least one transformer configured to provide an attention map based on image features from another searching block; generating image features of a second resolution using the first parallel module, wherein the first parallel module includes a second plurality of stacked searching blocks and one or more searching blocks of the second plurality of stacked searching blocks includes a plurality of convolution layers and at least one transformer configured to provide an attention map based on image features from a different searching block; and fusing one or more image features received from the first plurality of stacked searching blocks with one or more image features received from the second plurality of stacked searching blocks to output multiscale image features of the first resolution and multiscale image features of the second resolution.

DETAILED DESCRIPTION

NAS methods have achieved remarkable success in automatically designing efficient models for image classification. NAS has also been applied to improve efficiency of models for dense prediction tasks such as semantic segmentation and pose estimation. However, existing NAS methods for dense prediction either directly extend the search space designed for image classification or only search for a feature aggregation head. This lack of consideration to the specificity of dense prediction hinders the performance advancement of NAS methods compared to the best hand-crafted models.

In principle, dense prediction tasks require integrity of global context and high-resolution representations. The former is critical to clarify ambiguous local features at each pixel, and the latter is useful for accurate predictions of fine details, such as semantic boundaries and key point locations. However, these principles, especially HR representations, have not been the focus of prominent NAS algorithms for classification. Commonly, multi-scale features have been combined at the end of the network while recent approaches show that performance can be enhanced by putting multi-scale feature processing within the network backbone. In addition, multi-scale convolutional representations cannot provide a global outlook of the image since dense prediction tasks often come with high input resolution, while a network often covers a fixed receptive field. Therefore, global attention strategies such as SENet or non-local networks have been proposed to enrich image convolutional features. Transformers, widely used in natural language processing, have exhibited good results when combined with a convolutional neural network for image classification and object detection. However, the computational complexity associated with transformers increases quadratically with the number of pixels; thus, a transformer implementation is computationally expensive. In accordance with examples of the present disclosure, in-network multi-scale features and transformers are incorporated with NAS methods to obtain NAS enabled with dynamic task objectives and resource constraints.

In examples a dynamic down projection strategy is utilized to overcome issues associated with the computationally expensive costs associated with implementing transformers with image pixels. Accordingly, a lightweight and plug-and-play transformer architecture is described that that is combinable with convolutional neural architectures. In addition, to search a fused space of multi-scale convolutions and transformers, proper feature normalization, selection of fusion strategies, and balancing is needed. Accordingly, various model choices may be used that generalize and prefer multiple tasks based on the number of queries of the transformer.

In accordance with examples of the present disclosure, a super network also referred to as a “SuperNet” is first defined, where each layer of the SuperNet includes a multi-branch parallel module followed by a fusion module. The parallel module includes searching blocks with multiple resolutions, and the fusion module includes searching blocks of feature fusion that determine how features from different resolutions fuse. Based on the computational budget and task objectives, a fine-grained progressive shrinking search strategy can be used to efficiently prune redundant blocks in the network and channel in convolutions and transformer queries, resulting in an efficient model. In accordance with examples of the present disclosure, a transformer that is highly efficient and can be easily combined with convolutional networks for image and computer vision tasks is described. In accordance with examples of the present disclosure, a multi-resolution search space including both convolutions and transformers to model in-network multi-scale information and global contexts for dense prediction tasks is described. Thus, a transformer integrated into a resource-constrained NAS search space for image and computer vision tasks is described. In accordance with examples of the present disclosure, a resource-aware method of search that determines efficient architectures for different tasks is described.

FIG.1depicts a neural network system, also referred to as a transformer102, in accordance with examples of the present disclosure. The transformer102is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The transformer includes a projector110, an encoder104and a decoder106. Generally, both the encoder104and the decoder106are attention-based, i.e., both apply an attention mechanism (e.g., Multi-Head Self-Attention configuration) over their respective received inputs while transforming an input sequence. In some cases, neither the encoder nor the decoder includes any convolutional layers or any recurrent layers. The projector110uses a point-wise convolution (with Batch Normalization) to reduce the channel dimension of the feature map from c+dp(wherein c denotes the channel number of the input feature X, and dpdenotes the channel number of the positional map P) to a smaller dimension n, where n denotes the number of queries. The projector110may use bilinear interpolation to resize the spatial dimension of the feature map to s×s. That is, to reduce computational cost, the input feature X∈c×h×wis projected into a reduced size of n×s×s, by a projection function(⋅), where n denotes the number of queries and sxs is the reduced spatial size. Thus, the projection process can be represented by X′=(Concat(X, P)), where Concat denotes the concatenation operator, input sequence X′∈n×ssis the projected and flattened embedding, P∈dp×h×wis a positional encoding which compensates for the loss of spatial information during the self-attention process. When dp=2, P∈n×s2can be a two-dimensional positional map compensating for the loss of spatial information during the self-attention process. Compared with sinusoidal positional encoding and learned embeddings, the two-dimensional positional map P, which contains two channels (i.e., dp=2), is more efficient in terms of computational requirements for lightweight visual models. The two-dimensional position map can be obtained according to the following equations:

A 1×1 convolutional and bilinear interpolation may be performed to achieve the projection P(⋅) and inverse projection {tilde over (P)}(⋅) in the transformer102. The original image features X112may be divided into n tokens108to achieve a low-dimensional space. Each token108may be concatenated at116with the 2D positional map114to arrive at a projected feature118. That is, the input image feature X112is transformed into a set of n tokens X′, and each token in the set of n tokens X′ includes an s2dimensional semantic embedding with positional information. The projected features X′ may then be provided to the encoder104as queries, keys, and values Q, K, V∈n×s2.

where h is the number of heads, d is the hidden dimensions of the attended subspaces, and WiQ, WiK∈s2×d, WiV∈s2×d, WO∈hd×s2are learned embedding s (weights).

Using a residual connection, the output of the Multi-Head Self-Attentionconfiguration122is combined with the inputs to the Multi-Head Self-Attention122at an addition and normalization operation124. The output of the addition and normalization operation124is an encoder self-attention residual output which is provided to a position-wise feed-forward network126. The position-wise feed-forward network(⋅)126may include two linear transformations with a ReLU activation in between; the position-wise feed-forward network(⋅)126is applied to the attended features as(x)=max(0, xW1+b1)W2+b2, where the expansion ratio(⋅) is set to 4 for example, W1∈s2×4s2, W2∈4s2×s2, b1and b2denotes weights and a bias of the linear layers, respectively.

Accordingly, the encoder104can be represented by((Q, K, V)), where the token-wise attention A∈n×nis first calculated and linear transformation is then applied across the spatial-wise positions to obtain the global attended feature F. A residual connection from the Add & Norm operation124around the Feed-Forward Network126to the Add & Norm128is employed. The output of the encoder104is provided to the decoder106.

The decoder106follows a similar flow as the encoder104; the output from the encoder104is provided to the Multi-Head Self-Attention configuration130, where the Multi-Head Self-Attention configuration130also receives semantic queries S132. That is, Q, K, and V are provided to the Multi-Head Self-Attention configuration130. The Multi-Head Self-Attention configuration130uses the output of the encoder104as keys and values and the learnable semantic embeddings S∈n×s2(e.g., a set of n learnable s2-dimensional sematic embeddings) as queries. Using a residual connection, the output of the Multi-Head Self-Attentionconfiguration130is combined with the inputs to the Multi-Head Self-Attention130at the additional and normalization operation138to generate decoder self-attention residual output. The decoder self-attention residual output is provided to the position-wise feed-forward network(⋅) configuration136. A residual connection from the Add & Norm operation134around the feed-forward network136to the Add & Norm operation138is employed. The output of the decoder106is then projected back to the original feature size c×h×w by an inverse projection function(⋅) and then added to the image features X112. Because the image modeling is not a prediction task, and there are no temporal relationships between the semantic embedded queries, a first Multi-Head Attention configuration in a standard Transformer decoder (that is, a first Multi-Head Attention configuration that provides an input to the Multi-Head Attention configuration130) can be omitted from the decoder106.

Time complexity of the Multi-Head Self-Attention and the Feed-Forward Networks are O(4nds2+2n2d) and O(8ns4), where s2, d, and n are in the projected low-dimensional space. Since s2is a projected small spatial size, the overall time complexity (FLOPs) Oτ(n) of the transformer102is approximately linear with n2d. Accordingly, in some examples, the Transformer102may be utilized in a fine-grained search strategy to reduce and select an appropriate n to further make the Transformer102more efficient.

Non-limiting differences between the Transformer102and a standard Transformer include the use of the projection function(⋅) for learning self-attention in a low-dimensional space; using a two-dimensional positional map P rather than a sinusoidal positional encoding; the first Multi-Head Attention and the spatial encoding in the standard Transformer decoder are omitted; and the output of the encoder104is directly used as the keys and values of the decoder106with residual connections (e.g., a residual connection around the Multi-Head Self-Attentionconfiguration130).

In accordance with examples of the present disclosure,FIG.2depicts a multi-branch search space202for dense predictions that includes both multiscale features and global contexts while maintaining high-resolution representations throughout the neural network. The SuperNet204is a multi-branch network including a plurality of searching blocks210, where each searching block includes at least one convolutional layer214; in an example, the searching block210may also include a transformer212. The transformer212may be the same as or similar to the transformer102previously described in the present disclosure. Unlike previous searching methods for a specific task, the network searching network may be customized for various dense prediction tasks. The multi-branch search space may include a parallel module208and a fusion module206. In an example, the parallel module208and a fusion module206are configured alternatively. For example, a fusion module can be used after a parallel module to exchange information across multiple branches. In an example, the parallel module208and the fusion module206utilize the searching blocks210.

FIG.3depicts additional details of the multi-branch search space for dense predictions in accordance with examples of the present disclosure. As depicted inFIG.3, after one or more convolutional layers304reduce the feature solution, for example, to one quarter of the image size, low-resolution convolution branches are gradually added to high-resolution convolution branches using feature fusion through fusion modules306,314, etc. Multi-resolution branches are connected in parallel using the parallel modules, for example, parallel modules308,312,316, etc. The multi-branch features are concatenated together and connected to a final classification/regression layer at318.

The parallel module320, which may be the same as or similar to the parallel modules308,312,316, etc. generally obtains larger receptive fields and multi-scale features by stacking searching blocks in each branch. For example, a searching block334A may reside between feature maps322and324; a searching block334B may reside between the feature maps324and326. The searching blocks334A and334B may be the same or different. Feature maps322,324, and326are illustrative examples of higher-resolution feature maps. Similarly, a searching block334C may reside between feature maps328and330; a searching block334D may reside between the feature maps330and332. The searching blocks334C and334D may be the same or different. Searching blocks334A,334B,334C, and334D may be the same or different. Feature maps328,330, and332are illustrative examples of feature maps having a lower resolution than the feature maps322,324, and326. In examples, the parallel module320includes m∈[1,4] branches containing nc1, . . . ncmconvolutional layers with nw1, . . . nwmchannels in each branch. That is, a parallel module can be represented as [m, [nc1, . . . , ncm], [nw1, . . . , nwm]].

The fusion module336, which may be the same as or similar to the fusion modules306,314, etc. are utilized between two parallel modules with minand moutbranches to perform feature interactions between multiple branches using element-wise addition. For each output branch, neighboring input branches are fused using a searching block to unify feature map sizes. For example, an 8× output branch contains information of 4×, 8×, and 16× input branches. The high-to-low resolution feature transformation is implemented with a searching block and up-sampling. For example, searching blocks represented as arrows in the fusion module336may reside between feature maps338and334,338and340,342and340,342and344,342and348,346and344,346and348, and346and350. As in the parallel module, the searching blocks may be the same as each other or may be different from one another.

FIG.4depicts additional details of a searching block406in accordance with examples of the present disclosure. The searching block406may be the same as the searching block404in the parallel module and/or the searching block410in the fusion module. In examples, the searching block includes convolution layers412and at least one transformer430, where the number of convolutional channels and the number of queries/tokens in the at least one transformer are searchable parameters. In examples, the convolutional layers412in the searching block406are organized following an efficient structure of an inverted residual block, and the at least one transformer430is included to enhance global contexts. In some examples, the convolutional layers412may be different than or otherwise include a different configuration than that which is depicted inFIG.4. Similarly, in some examples, the searching block406may include a modified transformer that is different than the at least one transformer430depicted inFIG.4, or the at least one transformer430may be omitted in its entirety.

If c denotes the channel number of the input feature X and the spatial dimensions hxw is omitted for reasons of simplicity, the first layer414may be defined as a 1×1 point-wise convolution Co. The first layer is defined as a 1×1 pointwise C0∈c×3rcto expand the input feature to a high dimension having an expansion ratio of 3r using the convolution416,418, and420. The three depth-wise convolutional layers, C1424, C2422, C3426∈rcwith different kernel sizes of 3×3, 5×5, 7×7, are imposed on the three parts of the expended feature respectfully. The output of layers424,422, and426are then concatenated followed by a point-wise convolutional layer C′428∈3rc×cto reduce the number of channels to c′ (c′=c in the parallel module). At a same time, the Transformerwith n queries is applied to the input feature X to obtain global self-attention, which is then added to a final output. In this way, the Transformeris considered to be a residual path to enhance the global context within each searching block. The information flow in a searching block can be written as: X′=C4(Concat(C1(C0(X)1), C2(C0(X)2), C3(C0(X)3)))+(X), where C0(X), represents the i-th portion of the output of the first convolutional layer C0(X), as depicted inFIG.4. In examples, a stride of two in the convolutions C1, C2, C3and a half-size inverse projection(⋅) in the transformer are used for reducing a searching block. In this way, the whole SuperNet (e.g.,302FIG.3) is constructed by reduction searching block described herein, making such a model an easy fit for a limited computational budget by shrinking the depth-wise convolutional channels of C1, C2, C3and queries/tokens of Transformerwhile maintaining multi-scale and global information.

The SuperNet (e.g.,302FIG.3) is a multi-branch network including searching blocks, where each searching block may include a mixture of convolutional layers and a Transformer. Unlike previous searching methods for a specific task, the network for various dense prediction tasks may be customized to obtain an optimal feature combination for different tasks. For example, a resource-aware channel/query wise fine-grained search strategy may be used to explore the optimal feature combination for different tasks.

In examples, a progressive shrinking neural architecture search paradigm is used to generate light-weight models by discarding a portion of the convolutional channels and Transformer queries during training. In the searching block (e.g.,406), the 1×1 convolutional layers C0, C4are utilized to ensure that each cell has fixed input and output dimensions. In contrast, the interaction between channels in depth-wise convolutions C1, C2, C3can be minimized such that that unimportant channels can be easily removed in the search processes. For example, if a channel in C1is unimportant and removed, convolutions C0, C4can be adjusted to c×(3rc−1) and (3rc−1)×c′ respectively (wherein c and c′ represent the number of channels of convolutions C0, C4, respectively). Similarly, with a projection(⋅) and the inverse projection(⋅), the Transformermay be designed to include a variable number of queries and tokens. If a query is discarded, then the projections(⋅) and(⋅) can process (n−1)×s×s sized features in the low-dimensional space. Accordingly, tokens and features of both the transformer of the encoder and the transformer of the decoder are automatically scaled. As an example, a searching block (e.g.,406) may contain (3rc+n) learnable sub-layers, wherein c is the number of channels of the searching block406, r is the expansion ratio, and n is the number of tokens.

In examples, a factor α>0 can be learned jointly with the network weights to scale the output in each learnable sublayer of the search block (e.g.,406). The channels and queries having low importance can be progressively discarded while maintaining an overall performance of the searching block. In some examples, a resource-aware penalty on α may push other important factors to near-zero values. For example, the computational cost γ>0 for each sub-layer of the searching block (e.g.,406) is used to weight the penalty to fit for a limited computational budget:

Whereis as provided above; i is the index of sub-layers, n═ is the number of remaining queries (tokens), and γiis the computational cost of the ith sub-layer. Thus, γ may be a fixed value in the three depth-wise convolutions C1, C2, C3, while in the Transformerit is a dynamic value set according to the number of remaining queries. With the added resource-aware penalty term, the overall training loss is:

Where Ltaskdenotes the standard classification/regression loss with the weight decay term for a specific task, and λ denotes the coefficient of the L1 penalty term. The weight decay may help to constrain the value of the network weight to prevent it from being too large and making important factors a difficult to learn. Within several epochs as time intervals, sub-layers having an important factor that is less than a threshold ϵ can be removed and the statistics of Batch Normalization (BN) layers can be re-calibrated. If all tokens/queries of the Transformer are removed, the Transformer will degenerate into a residual path. When the search ends, the remaining structure can be used directly without the need for fine-tuning.

Based on resource-aware L1 regularization, an accuracy-efficiency trade-off for different amount of resource budges can be found. Considering that FLOPs is the most widely and easily used metric and approximated as the lower bound of the latency, FLOP may be used as a penalty weight. Other metrics can be applied similarly Moreover, the multi-branch SuperNet can be customized for different tasks during the search process. Different convolutional channels and Transformer tokens of different branches are retained for different tasks; thus, the optimal low-level/high-level and local/global feature combination for a specific task can be identified.

FIG.5depicts additional details of the multi-branch search space for dense predictions in accordance with examples of the present disclosure. In examples, the multi-branch search space includes a high-resolution convolution stream that is received at a first stage, and gradually adds high-to-low resolution streams one by one, forming new stages, and connecting multiresolution streams in parallel. As a result, the resolutions for the parallel streams of a later stage includes the resolutions from the previous stage, and an additional lower resolution. In accordance with examples of the present disclosure, a first fusion module503may receive, as input, a high-resolution convolution stream502, where the high-resolution convolution stream may be at a first resolution510. The first fusion module503may be the same as or similar to the fusion module306. The first fusion module503may add a high-to-low resolution stream corresponding to a second step or resolution512. For example, a searching block524, which is represented by an arrow and may be the same as or similar to a search block406(FIG.4), may initiate the convolution stream of the second resolution512.

A parallel module504, which may be the same as or similar to the parallel module308and/or320ofFIG.3, may stack searching blocks, represented by arrows, in each branch, where a first branch may correspond to a first resolution510and the second branch may correspond to the second resolution512. The searching blocks in the parallel module504may be the same as or similar to the search blocks406ofFIG.4. Another fusion module505, which may be the same as or similar to the fusion module336ofFIG.3, may exchange information across multi-resolution representations (e.g., features at a first resolution510and features at a second resolution512). Accordingly, the fusion module505may up-sample feature information from the second resolution512and fuse such information with the feature information from the first resolution510. Similarly, the fusion module505may down-sample feature information from the first resolution510and fuse such information with feature information from the second resolution512. Similar to the fusion module503, the fusion module505may add a high-to-low resolution stream corresponding to a third step or resolution514.

A parallel module506may be between the fusion module505and a fusion module507. The fusion module507may up-sample feature information from the second resolution512and fuse such information with the feature information from the first resolution510. Similarly, the fusion module507may down-sample feature information from the first resolution510and fuse such information with feature information from the second resolution512and feature information up-sampled from the third resolution514. The fusion module507may down-sample feature information from the second resolution512and fuse such information with feature information from the third resolution514Similar to the fusion modules503and505, the fusion module507may add a high-to-low resolution stream corresponding to a fourth step or resolution516. In examples, the fusion module507is the same as or similar to the fusion module314ofFIG.3.

A parallel module508may reside between the fusion module507and a fusion module509. The fusion module509may operate in a similar manner as the fusion module507, fusing feature information from various resolutions and adding a high-to-low resolution stream corresponding to a fifth step or resolution518. In examples, the number of parallel modules and fusion modules may be different than that which is depicted inFIG.3,FIG.4, and/orFIG.5. In examples, there may be more or less fusion modules and feature modules than that which is depicted.

In examples, the searching blocks represented by arrows may be a searching block532A and/or532B, where the searching block532A may be the same as or similar to the searching block406(FIG.4), which may include convolution layers412and a transformer430. The searching block532A, in some examples, may perform a low-to-high resolution feature transformation; in some examples, the resolution of the feature transformation may remain the same. In some examples, the searching block implementing a high-to-low resolution feature transformation may implement a searching block532B, where the searching block532A may be the same as or similar to the searching block406(FIG.4), which may include convolution layers412and a transformer430. The searching block532B may be referred to as a reduction searching block.

FIG.6depicts details of a method600for generating attention maps using a transformer in accordance with examples of the present disclosure. A general order for the steps of the method600is shown inFIG.6. Generally, the method600starts at602and ends at618. The method600may include more or fewer steps or may arrange the order of the steps differently than those shown inFIG.6. The method600can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. In examples, aspects of the method600are performed by one or more processing devices, such as a computer or server. Further, the method600can be performed by gates or circuits associated with a processor, Application Specific Integrated Circuit (ASIC), a field programmable gate array (FPGA), a system on chip (SOC), a neural processing unit, or other hardware device. Hereinafter, the method600shall be explained with reference to the systems, components, modules, software, data structures, user interfaces, etc. described in conjunction withFIGS.1-5.

The method starts at602, where flow may proceed to604. At604, one or more input feature maps may be received. To reduce computational cost, the input feature X is projected into a reduced size by a projection function(⋅) at606. Compared with sinusoidal positional encoding and learned embeddings, the two-dimensional positional map P, which contains two channels, is more efficient in terms of computational requirements for lightweight visual models.

The encoder of the transformer may include a Multi-Head Self-Attention(⋅) configuration, which allows the encoder to jointly attend to information at different positions. Further, using a residual connection layer, the output of the Multi-Head Self-Attention configuration is combined with the inputs to the Multi-Head Self-Attentionto generate an encoder self-attention residual output. The encoder self-attention residual output is provided to a feed-forward network. At608, an output from the encoder is provided to a Multi-Head Self-Attention configurationof a decoder, where the Multi-Head Self-Attention configurationof the decoder also receives semantic queries at610. That is, the keys K and values V, are provided to the Multi-Head Self-Attention configurationof the decoder from the encoder portion of the transformer; the queries Q are learnable semantic embeddings S∈n×s2(e.g., a set of n learnable s2-dimensional sematic embeddings). The decoder may then obtain an output based on Q, K, and V at612. That is, a Multi-Head Self-Attention configurationuses the output of the encoder F as keys and values and the learnable semantic embeddings as queries. Using a residual connection layer, the output of the Multi-Head Self-Attentionconfiguration of the decoder is combined with the inputs to the Multi-Head Self-Attentionto generate decoder self-attention residual output. The output is provided to a position-wise feed-forward network(⋅) configuration. A residual connection feeds the input of the position-wise feed-forward network around the feed-forward network to an addition and normalization operation. The output of the decoder is then projected back to the original feature size c×h×w by an inverse projection function(⋅) at614to acquire attention features. The features may then be added to the image features X. In examples, the output of the Transformer may be added to a convolutional layer within a searching block (e.g.,406) as previously described. The method600may end at618.

FIG.7depicts details of a method700for performing a network architecture search in accordance with examples of the present disclosure. A general order for the steps of the method700is shown inFIG.7. Generally, the method700starts at702and ends at716. The method600may include more or fewer steps or may arrange the order of the steps differently than those shown inFIG.7. The method700can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. In examples, aspects of the method700are performed by one or more processing devices, such as a computer or server. Further, the method700can be performed by gates or circuits associated with a processor, Application Specific Integrated Circuit (ASIC), a field programmable gate array (FPGA), a system on chip (SOC), a neural processing unit, or other hardware device. Hereinafter, the method700shall be explained with reference to the systems, components, modules, software, data structures, user interfaces, etc. described in conjunction withFIGS.1-6.

The method starts at702, where flow may proceed to704. At704, a SuperNet is setup or otherwise configured. The SuperNet may be the same as or similar to the SuperNet302(FIG.3) and generally includes one or more parallel modules and one or more fusion modules, where each of the parallel modules and each of the fusion modules may include the searching block as previously described (e.g.,406FIG.4). Each searching block can include convolution layers and a transformer as previously described according to the examples of the present disclosure. In examples, convolutional layers of the SuperNet may reduce the spatial dimension of image features. For example, the spatial dimension of image features may be reduced by a factor of four. Starting at a high-resolution branch of the SuperNet, at706, image features of a first resolution may be generated using a first plurality of stacked searching blocks in a first parallel module for example. At708, image features of a second resolution may be generated by the first parallel module. For example, the first parallel module may include a plurality of stacked searching blocks at a first resolution level and a plurality of stacked searching blocks at a second resolution level. Thus, image features of the first resolution may be generated by the plurality of stacked searching blocks, and image features of the second resolution may be generated by the second plurality of stacked searching blocks. At710, a fusion module may generate multiscale image features of the first resolution and multiscale image features of the second resolution by fusing image features of the first resolution and image features of the second resolution. In examples, a searching block in the fusion module may adjust a spatial dimension, or resolution, of the image features via up-sampling or down-sampling depending on which branch the fusion module resides. For example, a high-to-low resolution image feature transformation may be realized by a reduction searching block while the low-to-high resolution feature transformation may be realized with a different searching block. Accordingly, an output branch of the fusion module may include information from a plurality of branches of the SuperNet. In some examples, the SuperNet may be pruned at712. That is, a portion of the convolutional channels and transformer queries of some searching blocks may be discarded as previously described. The method700may end at714.

FIG.8is a block diagram illustrating physical components (e.g., hardware) of a computing system800with which aspects of the disclosure may be practiced. The computing system components described below may be suitable for the computing and/or processing devices described above. In a basic configuration, the computing system800may include at least one processing unit802and a system memory804. Depending on the configuration and type of computing device, the system memory804may comprise, but is not limited to, volatile storage (e.g., random-access memory (RAM)), non-volatile storage (e.g., read-only memory (ROM)), flash memory, or any combination of such memories.

The system memory804may include an operating system805and one or more program modules806suitable for running software application820, such as one or more components supported by the systems described herein. As examples, system memory804may include one or more of the following: transformer821, projector822, encoder823, decoder824, SuperNet825, parallel module826, fusion module827, searching block828, and/or convolution configuration829. The transformer821may be the same as or similar to the transformer102previously described. The projector822may be the same as or similar to the projector110previously described. The encoder823may be the same as or similar to the transformer102previously described. The decoder824may be the same as or similar to the decoder106previously described. The SuperNet825may be the same as or similar to the SuperNet302previously described. The parallel module826may be the same as or similar to the parallel module320previously described. The fusion module827may be the same as or similar to the fusion module336previously described. The searching block828may be the same as or similar to the searching block406previously described. The convolution configuration829may be the same as or similar to the convolutional layers412as previously described. One or more of the components depicted in the system memory804may include one or more of the other components depicted in the system memory804. For example, the transformer821may include an encoder823and a decoder824. The operating system805, for example, may be suitable for controlling the operation of the computing system800.

Furthermore, examples of the disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program and is not limited to any particular application or system. This basic configuration is illustrated inFIG.8by those components within a dashed line808. The computing system800may have additional features or functionality. For example, the computing system800may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated inFIG.8by a removable storage device809and a non-removable storage device810.

As stated above, a number of program modules and data files may be stored in the system memory804. While executing on the processing unit802, the program modules806(e.g., software applications820) may perform processes including, but not limited to, the aspects, as described herein. Other program modules that may be used in accordance with aspects of the present disclosure may include electronic mail and contacts applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided programs, etc.

The computing system800may also have one or more input device(s)812such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or swipe input device, etc. The one or more input device812may include an image sensor. The output device(s)814such as a display, speakers, a printer, etc. may also be included. The aforementioned devices are examples and others may be used. The computing system800may include one or more communication connections816allowing communications with other computing devices/systems850. Examples of suitable communication connections816include, but are not limited to, radio frequency (RF) transmitter, receiver, and/or transceiver circuitry; universal serial bus (USB), parallel, and/or serial ports.

FIGS.9A-9Billustrate a mobile computing device900, for example, a mobile telephone, a smart phone, wearable computer (such as a smart watch), a tablet computer, a laptop computer, and the like, with which examples of the disclosure may be practiced. In some examples, the mobile computing device900may utilize a trained search space and/or trained model to perform one or more tasks, such as an image classification task. In other example, the mobile computing device900may provide information to system, such as the computing system800, and receiving information from the computing system800. In some examples, the mobile computing device900may be the same as or similar to the computing system800. In some respects, the client may be a mobile computing device. With reference toFIG.9A, one aspect of a mobile computing device900for implementing the aspects is illustrated. In a basic configuration, the mobile computing device900is a handheld computer having both input elements and output elements. The mobile computing device900typically includes a display905and one or more input buttons910that allow the user to enter information into the mobile computing device900. The display905of the mobile computing device900may also function as an input device (e.g., a touch screen display).

If included, an optional side input element915allows further user input. The side input element915may be a rotary switch, a button, or any other type of manual input element. In alternative aspects, mobile computing device900may incorporate greater or fewer input elements. For example, the display905may not be a touch screen in some embodiments.

In yet another alternative embodiment, the mobile computing device900is a portable phone system, such as a cellular phone. The mobile computing device900may also include an optional keypad935. Optional keypad935may be a physical keypad or a “soft” keypad generated on the touch screen display.

In various embodiments, the output elements include the display905for showing a graphical user interface (GUI), a visual indicator920(e.g., a light emitting diode), and/or an audio transducer925(e.g., a speaker). In some aspects, the mobile computing device900incorporates a vibration transducer for providing the user with tactile feedback. In yet another aspect, the mobile computing device900incorporates input and/or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., a HDMI port) for sending signals to or receiving signals from an external device.

FIG.9Bis a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, the mobile computing device900can incorporate a system (e.g., an architecture)902to implement some aspects. In one embodiment, the system902is implemented as a “smart phone” capable of running one or more applications (e.g., browser, e-mail, calendaring, contact managers, messaging clients, games, media clients/players, and other apps). In some aspects, the system902is integrated as a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

One or more application programs966may be loaded into the memory962and run on or in association with the operating system964. Examples of the application programs include phone dialer programs, e-mail programs, imaging programs, multimedia programs, video programs, word processing programs, spreadsheet programs, Internet browser programs, messaging programs, maps programs, and so forth. The system902also includes a non-volatile storage area968within the memory962. The non-volatile storage area968may be used to store persistent information that should not be lost if the system902is powered down. The application programs966may use and store information in the non-volatile storage area968, such as e-mail or other messages used by an e-mail application, and the like. A synchronization application (not shown) also resides on the system902and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in the non-volatile storage area968synchronized with corresponding information stored at the host computer. As should be appreciated, other applications may be loaded into the memory962and run on the mobile computing device900described herein.

The system902has a power supply970, which may be implemented as one or more batteries. The power supply970might further include an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the batteries.

The system902may also include a radio interface layer972that performs the function of transmitting and receiving radio frequency communications. The radio interface layer972facilitates wireless connectivity between the system902and the “outside world,” via a communications carrier or service provider. Transmissions to and from the radio interface layer972are conducted under control of the operating system964. In other words, communications received by the radio interface layer972may be disseminated to the application programs966via the operating system964, and vice versa.

The visual indicator920may be used to provide visual notifications, and/or an audio interface974may be used for producing audible notifications via the audio transducer925. In the illustrated embodiment, the visual indicator920is a light emitting diode (LED) and the audio transducer925is a speaker. These devices may be directly coupled to the power supply970so that when activated, they remain on for a duration dictated by the notification mechanism even though the processor960and other components might shut down for conserving battery power. The LED may be programmed to remain on indefinitely until the user takes action to indicate the powered-on status of the device. The audio interface974is used to provide audible signals to and receive audible signals from the user. For example, in addition to being coupled to the audio transducer925, the audio interface974may also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. In accordance with embodiments of the present disclosure, the microphone may also serve as an audio sensor to facilitate control of notifications, as will be described below. The system902may further include a video interface976that enables an operation of an on-board camera930to record still images, video stream, and the like.

A mobile computing device900implementing the system902may have additional features or functionality. For example, the mobile computing device900may also include additional data storage devices (removable and/or non-removable) such as, magnetic disks, optical disks, or tape. Such additional storage is illustrated inFIG.9Bby the non-volatile storage area968.

Data/information generated or captured by the mobile computing device900and stored via the system902may be stored locally on the mobile computing device900, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer972or via a wired connection between the mobile computing device900and a separate computing device associated with the mobile computing device900, for example, a server computer in a distributed computing network, such as the Internet. As should be appreciated such data/information may be accessed via the mobile computing device900via the radio interface layer972or via a distributed computing network Similarly, such data/information may be readily transferred between computing devices for storage and use according to well-known data/information transfer and storage means, including electronic mail and collaborative data/information sharing systems.

FIG.10illustrates one aspect of the architecture of a system for processing data received at a computing system from a remote source, such as a personal computer1004, tablet computing device1006, or mobile computing device1008, as described above. The personal computer1004, tablet computing device1006, or mobile computing device1008may include one or more applications. Content at a server device1002may be stored in different communication channels or other storage types.

One or more of the previously described program modules or software applications804(FIG.8) may be employed by server device1002and/or the personal computer1004, tablet computing device1006, or mobile computing device1008, as described above. For example, the server device1002may include a transformer1021and/or a SuperNet1025; the SuperNet1025may be included in an untrained state and/or after training, as a network model trained for a specific task, such as image classification for example.

The server device1002may provide data to and from a client computing device such as a personal computer1004, a tablet computing device1006and/or a mobile computing device1008(e.g., a smart phone) through a network1015. By way of example, the computer system described above may be embodied in a personal computer1004, a tablet computing device1006and/or a mobile computing device1008(e.g., a smart phone). Any of these embodiments of the computing devices may obtain content from the store1016, in addition to receiving graphical data useable to be either pre-processed at a graphic-originating system, or post-processed at a receiving computing system.

The present disclosure relates to a search space and systems and methods for obtaining and searching a search space according to at least the examples provided in the sections below:

(A1) In one aspect, some examples include a search space comprising a first parallel module including a first plurality of stacked searching blocks and a second plurality of stacked searching blocks, wherein the first plurality of stacked searching blocks is configured to output first feature maps of a first resolution and the second plurality of stacked searching blocks is configured to output second feature maps of a second resolution; a fusion module including a plurality of searching blocks, wherein the fusion module is configured to generate multiscale feature maps by fusing one or more feature maps of the first resolution received from the first parallel module with one or more feature maps of the second resolution received from the first parallel module, and wherein the fusion module is configured to output the multiscale feature maps and output third feature maps of a third resolution; and a second parallel module configured to receive the multiscale feature maps and the third feature maps of the third resolution from the fusion module, and output fourth feature maps of the first resolution, fifth feature maps of the second resolution, and sixth feature maps of the third resolution.

(A2) In some examples of A1, at least one searching block of the plurality of searching blocks of the fusion module is configured to down-sample feature maps, and at least one searching block of the first plurality of searching blocks of the fusion module is configured to up-sample feature maps.

(A3) In some examples of A1-A2, one or more searching blocks of the first plurality of stacked searching blocks includes a transformer configured to provide an attention map based on feature maps received from another searching block of the first plurality of stacked searching blocks.

(A4) In some examples of A1-A3, one or more searching blocks of the first plurality of stacked searching blocks includes a plurality of convolution layers arranged in a depth-wise manner, each convolution layer of the plurality of convolution layers having a different kernel size.

(A5) In some examples of A1-A4, the first resolution is greater than the second resolution.

(A6) In some examples of A1-A5, the search space includes a second fusion module including a second plurality of searching blocks, wherein the second fusion module is configured to generate multiscale feature maps of the second resolution by combining a down-sampled feature map received from the second parallel module with an up-sampled feature map received from the second parallel module.

(A7) In some examples of A1-A6, the fusion module is configured to fuse feature maps from searching blocks of three different resolutions.

(A8) In some examples of A1-A7, the search space includes another fusion module configured to receive a convolution stream and output feature maps of the first resolution to the first parallel module and output feature maps of the second resolution to the first parallel module.

In yet another aspect, some examples include a computing system including one or more processors and memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for generating a search space according to any one of A1-A8 described above.

In yet another aspect, some examples include a non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of a storage device, the one or more programs including instructions for generating a search space according to any one of A1-A8 described above.

The present disclosure relates to a search space and systems and methods for obtaining and searching a search space according to at least the examples provided in the sections below:

(B1) In one aspect, some examples include a search space comprising: a first branch including a first plurality of stacked searching blocks for image features of a first resolution, one or more searching blocks of the first plurality of stacked searching blocks including a plurality of convolution layers and at least one transformer configured to provide an attention map based on image features from another searching block of the first branch; a second branch including a second plurality of stacked searching blocks for image features of a second resolution, one or more searching blocks of the second plurality of stacked searching blocks including a plurality of convolution layers and at least one transformer configured to provide an attention map based on image features from another searching block of the second branch; and a fusion module configured to fuse image features output by the one or more searching blocks of the first plurality of stacked searching blocks and image features output by the one or more searching blocks of the second plurality of stacked searching blocks, wherein the fusion module is configured to output image features of the first resolution and image features of the second resolution.

(B2) In some examples of B1, the fusion module is configured to initiate a third branch and output image features of a third resolution.

(B3) In some examples of B1-B2, the first resolution is greater than the second resolution.

(B4) In some examples of B1-B3, the fusion module includes a searching block configured to down-sample image features of the first branch and up-sample image features of the third branch, the fusion module configured to generate multiscale image features by fusing the down-sampled image features and the up-sampled image features to output multiscale image features of the second resolution.

(B5) In some examples of B1-B4, one or more searching blocks of the first plurality of stacked searching blocks includes a plurality of convolution layers arranged in a depth-wise manner, each convolution layer of the plurality of convolution layers having a different kernel size.

(B6) In some examples of B1-B5, the search space includes a third branch including a third plurality of stacked searching blocks for image features of a third resolution, wherein one or more searching blocks of the third plurality of stacked searching blocks includes a transformer.

In yet another aspect, some examples include a computing system including one or more processors and memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for generating a search space according to any one of B1-B6 described above.

In yet another aspect, some examples include a non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of a storage device, the one or more programs including instructions for generating a search space according to any one of B1-B6 described above.

The present disclosure relates to systems and methods for searching a search space according to at least the examples provided in the sections below:

(C1) In one aspect, some examples include a method of searching a search space. The method may include generating image features of a first resolution using a first parallel module including a first plurality of stacked searching blocks, wherein one or more searching blocks of the first plurality of stacked searching blocks includes a plurality of convolution layers and at least one transformer configured to provide an attention map based on image features from another searching block; generating image features of a second resolution using the first parallel module, wherein the first parallel module includes a second plurality of stacked searching blocks and one or more searching blocks of the second plurality of stacked searching blocks includes a plurality of convolution layers and at least one transformer configured to provide an attention map based on image features from a different searching block; and fusing one or more image features received from the first plurality of stacked searching blocks with one or more image features received from the second plurality of stacked searching blocks to output multiscale image features of the first resolution and multiscale image features of the second resolution.

(C2) In some examples of C1, the method includes generating down-sampled image features of the second resolution using a searching block that receives image features from a searching block of the first plurality of stacked searching blocks.

(C3) In some examples of C1-C2, the method includes generating up-sampled image features of the second resolution using a searching block that receives image features from a searching block of a third plurality of stacked searching blocks.

(C4) In some examples of C1-C3, the method includes generating, by a fusion module, multiscale image features of a third resolution.

(C5) In some examples of C1-C4, at least one searching block of the first parallel module includes a plurality of depth-wise convolution layers, each convolution layer of the plurality of depth-wise convolution layers generating an output using a different kernel size.

(C6) In some examples of C1-05, the first resolution is greater than the second resolution.

In yet another aspect, some examples, include a computing system including one or more processors and memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described herein (e.g., C1-C6 described above).

In yet another aspect, some examples include a non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods described herein (e.g., C1-C6 described above).