Exploring Stable Meta-optimization Patterns via Differentiable Reinforcement Learning for Few-shot Classification

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 09:54:33

Abstract: Existing few-shot learning methods generally focus on designing exquisite structures of meta-learners for learning task-specific prior to improve the discriminative ability of global embeddings. However, they often ignore the importance of learning stability in meta-training, making it difficult to obtain a relatively optimal model. From this key observation, we propose an innovative generic differentiable Reinforcement Learning (RL) strategy for few-shot classification. It aims to explore stable meta-optimization patterns in meta-training by learning generalizable optimizations for producing task-adaptive embeddings. Accordingly, our differentiable RL strategy models the embedding procedure of feature transformation layers in meta-learner to optimize the gradient flow implicitly. Also, we propose a memory module to associate historical and current task states and actions for exploring inter-task similarity. Notably, our RL-based strategy can be easily extended to various backbones. In addition, we propose a novel task state encoder to encode task representation, which fully explores inner-task similarities between support set and query set. Extensive experiments verify that our approach can improve the performance of different backbones and achieve promising results against state-of-the-art methods in few-shot classification. Our code is available at an anonymous site: https://anonymous.4open.science/r/db8f0c012/.

Main paper: 
 introduction: 
 Few-shot learning [1, 18, 30, 42, 55] aims to explore generalizable visual knowledge from seen classes in training and then utilize the learned knowledge to correctly recognize unseen classes where only a few labeled samples are available. Few-shot learning has attracted ever-increasing interest in industrial and academic communities due to its various applications.Generally, existing methods [20, 28, 32, 53] often rely on designing exquisite structures of meta-learner for generating discriminative global embeddings of images to achieve promising performance. Recently, exploring task-specific prior for meta-tasks has been proven as an efficient way to generate discriminative global embeddings. To this end, existing methods [5, 36, 55] usually adopt task-adapting strategies to model the task-specific prior. Ye et al. [55] modeled the task-adapting strategy as non-linear transformations on the support set. Qiao et al. [36] formulated the task-adapting process as a quadratic programming problem on sample-wise distance constraints. Baik et al. [5] proposed to implicitly guide the embedding procedure via optimizations of task-specific loss functions. Despite the improvement of these methods, the meta-training process still suffers from severe problems, such as high variance [4] and noisy gradients [43] . These problems can lead to unstable optimization patterns in meta-training and cause wrong optimization direction, finally affecting the performance, as illustrated in Figure 1(a) . Figure 2 : The details of our approach. We describe the transaction from time step 𝑡 to 𝑡 + 1. The policy module, consisting of a GRU cell with a FC layer, takes the state of the current time step 𝑡 as input and outputs the parameters of the transformation layer (i.e., scale 𝛾 and shift 𝛽) as action A 𝑡 . The state of the current time step 𝑡 is the concatenation of the current task state E 𝑡 and previous action A 𝑡 -1 . The memory module enables rapid storage and retrieval of associations between task state E 𝑡 and hidden state h 𝑡 of action A 𝑡 .Prior approaches of reinforcement learning, including training sample augment [9] and spatial attention enhancement [17] , have proven their effectiveness in improving the performance of few-shot learning tasks. However, these methods mainly straightforwardly utilize RL in the pre-training stage, and they often suffer from unstable training and huge computational cost. This is because they model the environment as a non-differentiable black box, resulting in the training process requiring lots of trials and errors. Moreover, black-box RL is prone to underfit on unseen tasks outside of distribution [29] . For the meta-training stability, Anantha et al. [3] modeled the meta-learner as the agent of RL to learn the task-adaptive loss surface of base-learner for exploiting stable loss patterns. However, Anantha et al. [3] aims to stabilize the inner loop. In contrast, their approach is still limited to the outer loop since the environment lacks differentiability. Applying RL with a differentiable environment is proven to be efficient in robotics [29] , control [14] , and navigation [10] , by the ability to stabilize optimization process and improve generalization on unseen tasks. This inspires us to utilize the pros of differentiable RL to overcome the flaws of black-box RL-based approaches (e.g., optimization instability, and huge computational cost) in few-shot learning.Highly inspired by differentiable RL [14, 29] , we propose a generic differentiable RL-based meta-optimization strategy for fewshot classification (named Reinforced Meta-Optimization, RMO), as shown in Figure 1 (b). Our method aims to explore stable metaoptimization patterns in meta-training for improving the discriminative ability of task-adaptive embeddings. Hence, we design a differentiable RL policy to model the embedding procedure of transformation layers in meta-learner for implicitly optimizing the gradient flow. Specifically, we model the policy as a Gated Recurrent Unit (GRU) to learn generalizable optimization patterns across metatasks. To this end, we propose a memory module, modeled as a Differentiable Neural Dictionary (DND), to associate historical and current task states and actions for exploring inter-task similarity and boosting the learning of policy. In addition, we propose a novel task state encoder to encode task state for fully exploring inner-task similarity between supports and queries. We conduct experiments with various backbones and classifiers on three popular datasets. The results verify that our method can consistently improve the performance on different backbones and achieve promising results against state-of-the-art transductive methods in few-shot classification.Overall, the contributions are summarized as follows:(1) We propose a generic differentiable RL-based few-shot learning strategy to explore stable meta-optimization patterns for meta-training.  
related work 2.1 few-shot classification: 
 Meta-learning has been proven to be an effective way to address the few-shot classification problem [18, 24, 30, 57, 59 ]. These methods often train a meta-learner using diverse meta-tasks sampled from a task distribution to capture common patterns across the tasks. The design of a meta-learner can include learning task-adapted representations [24, 30, 55] , exploring inner-task correlations [18, 36] , and utilizing text-modal knowledge [32, 51] . Introducing transformation layers into the network has proven efficient in learning task-specific representations. These methods include feature map modulation [30, 46] , non-linear adaptation [55] , task-level projection [57] , and learning task-specific loss functions [5] . While these methods always tend to design complicated structures of meta-learner [17] , they ignore the importance of stable optimization patterns in meta-training. These methods often suffer from overfitting in meta-training and need careful parameter tuning [30] .MAML [12] is a powerful algorithm introducing meta-learning into diverse problems, including few-shot classification. It aims to learn optimizations to adapt to unseen tasks with few available training samples. However, MAML-based methods suffer from some problems, including training instability [4] , permutation sensitivity [54] , and memorization overfitting [56] . Solutions for MAML's shortcomings have been well studied. Ye et al. [54] proposed to initialize the classification weights with one single weight vector to overcome the permutation sensitivity problem in few-shot classification. Simon et al. [43] proposed to scale the noisy gradients of meta-learner via low-rank approximation to stabilize the meta-training. However, these methods mainly focus on improving the generic meta-learning algorithms. In few-shot learning, the stable optimization patterns in meta-training need further studying.Reinforcement Learning has been proven effective in addressing the few-shot learning problem. Chu et al. [9] proposed to utilize maximum-entropy RL to learn the cropping trajectories of training images. Hong et al. [17] proposed to utilize the RL strategy to optimize spatial attention of feature maps. However, these methods only utilized the RL strategy into the pre-training stage, and these methods are still limited by the shortcomings of RL (e.g. unstable training and huge computational cost). On training stability of fewshot learning, Anantha et al. [3] proposed to learn the scale of gradients of base-learner task-dependently via RL to explore stable loss patterns in meta-training. However, this work only studied the optimization stability of the inner loop in meta-training, and for the outer loop, this method is still limited by the shortcomings of RL since the environment is modeled as a black box. 
method: 
 Our approach (as shown in Figure 2 ) mainly consists of a policy module, a memory module, and a task state encoder. We describe each module and the optimization strategy of our whole method in the following subsections.Problem statement. In the meta-training phase, the model faces a set of tasks. Each task is represented as a N -way K-shot problem with N classes sampled from the seen class set S (i.e., classes in training set) and K labeled samples per class, which composes the support set. Besides, M unlabeled samples are sampled per class as the query set. We denote the support set as S = {x i , y i } NK i=1 and the query set as Q = {x i } NK+NM i=NK+1 with the instance x i and the label y i ∈ S. In validation and evaluation, the N classes are sampled from the unseen class set U (i.e., classes in the validation set and testing set). Note that S∩U = ∅. The goal is to find an optimal function 𝑓 that correctly classifies the test sample x 𝑡𝑒𝑠𝑡 via ŷ𝑡𝑒𝑠𝑡 = 𝑓 (x 𝑡𝑒𝑠𝑡 ; S) ∈ U.Model design. First, we describe how the Feature-wise Linear Modulation (FiLM) layer [33] transforms the feature map. The learnable parameters of a FiLM layer include scale 𝛾 and shift 𝛽, where 𝛾, 𝛽 ∈ R 𝑐 . We split the backbone network G into two parts, where G 1 is the unchanged part and G 2 is the part where a FiLM layer is inserted after the last Batch Normalization (BN) layer in each convolution block. An intermediate feature mape ′ x = G 1 (x) of an input image x is first obtained. Denote z ∈ R ℎ * 𝑤 * 𝑐 is the activa- tion of e ′x produced by the BN layer in G 2 . Then z is modulated as: ẑℎ,𝑤,𝑐 = 𝛾 𝑐 * 𝑧 ℎ,𝑤,𝑐 + 𝛽 𝑐 , where ẑℎ,𝑤,𝑐 ∈ ẑ, 𝑧 ℎ,𝑤,𝑐 ∈ z, and 𝑐 denotes the number of channels. This modulation is applied in each block of G 2 . At last, the part G 2 outputs the embedding e x ∈ R 𝑑 of input image x, denoted as e x = G 2 (e ′x ; 𝛾, 𝛽). Denote 𝜃 G as the set of all learnable parameters of G. Different from previous methods [30, 39] , we model the learning process of parameters 𝛾 and 𝛽 with differentiable RL to optimize the gradient flow of the meta-learner. We describe how we model this process with the key elements of differentiable RL in the following. State. RL is an iterative process where the policy takes the current state as input and picks up an action in each time step. Denote 𝑡 = 1, • • • , T as the current time step and T as the total number of time steps. The next state S 𝑡 +1 is obtained from a differentiable transition function: S 𝑡 +1 = Φ(S 𝑡 , A 𝑡 ). We model the state S 𝑡 as the concatenation of current task state E 𝑡 and previous action A 𝑡 -1 . We depict how we obtain the task state E 𝑡 via task state encoder in Section 3.3. Task State Encoder. Action. In each time step, the policy outputs an action based on the current task state. We model the action A 𝑡 as the concatenation of parameters 𝛾 𝑡 and 𝛽 𝑡 . Here 𝛾 𝑡 and 𝛽 𝑡 denotes the values of 𝛾 and 𝛽 in time step 𝑡, respectively. We depict how the action A 𝑡 is picked up in Section 3.1. Policy Module. Reward. The environment rewards the quality of the action to guide the RL model. The reward 𝑟 𝑡 is calculated with a differentiable function: 𝑟 𝑡 = Ψ(S 𝑡 , A 𝑡 ). We model the current reward 𝑟 𝑡 as the classification loss calculated by the classifier. We depict how the reward 𝑟 𝑡 is calculated in Section 3.4. Optimization Strategy.We model the policy module as a minimal gated unit [60] , a variant of GRU with one single forget gate, to improve efficiency and reduce the number of parameters. In each time step 𝑡, the embeddings e 𝑡 x , x ∈ S ∪ Q are first transformed into the task state E 𝑡 , then concatenated with previous action A 𝑡 -1 to form input state S 𝑡 . Define 𝛾 0 = 1 and 𝛽 0 = 0, where 1 and 0 are two vectors filled with 1 and 0. The GRU cell takes S 𝑡 as input and outputs the action via the following equations:A 𝑡 = {𝛾 𝑡 , 𝛽 𝑡 } U n p u b l i s h e d w o r k i n g d r a f t . N o t f o r d i s t r i b u t i o n .EQUATIONwhere 𝜎 (•) and tanh(•) denote the sigmoid and hyperbolic tangent activation function, respectively, and ⊙ denotes the Hadamard product. Then h 𝑡 is taken by a fully connected layer without bias to get the mean 𝜇 𝑡 : 𝜇 𝑡 = W 𝑢 •h 𝑡 . The action A 𝑡 is sampled from the normal distribution: A 𝑡 ∼ N (𝜇 𝑡 , 1). The value of h 0 is initialized to 0 for each task.Inspired by the idea of episodic control [34, 40] , we model the memory module as a DND to explore inter-task similarity, enabling rapid storage and retrieve of associations between task state E 𝑡 and hidden state h 𝑡 of action A 𝑡 . The key and value of DND denote task state E 𝑡 and hidden state h 𝑡 , respectively. Then the new h 𝑡 is calculated using the following equations:EQUATIONwhere h 𝑒𝑝 denotes the hidden state from DND: h 𝑒𝑝 = 𝐷𝑁 𝐷 E ′ 𝑡 , and E ′ 𝑡 denotes the matched key of current E 𝑡 using a distance metric. We choose E ′ 𝑡 using top-1 cosine similarity with storedE 𝑡 : 𝑑 (E ′ 𝑡 , E 𝑡 ) = E ′ 𝑡 •E 𝑡 ∥E ′ 𝑡 ∥ ∥E 𝑡 ∥ .Finally, h 𝑡 and E 𝑡 are stored to DND: 𝐷𝑁 𝐷 [E 𝑡 ] = h 𝑡 . Denote 𝜃 𝑝 as the set of all learnable parameters of policy module and D as the number of memory units in DND.The architecture of the task state encoder is shown in Figure 3 . Unlike previous methods [30, 39] that only consider inner-task information from the support set, we utilize a single-head self-attention module to explore inner-task similarities between support set and query set. Here, we omit time step 𝑡 for simplicity. First, we obtain class representations by averaging the support samples in each class: ē𝑐 = 1 K (x 𝑖 ,y 𝑖 ) ∈S 𝑐 e x 𝑖 , where S 𝑐 denotes the support samples in 𝑐-th class, and𝑐 = 1, • • • , N . Define Se = {ē 𝑐 } N𝑐=1 and Q e = e x 𝑖 x 𝑖 ∈Q . We construct the query, key, and value using the following equation:EQUATIONAlgorithm 1 Training procedure of our approach.Require: Seen class set S, classifier 𝑓 𝑚 1: while training do 2:Sample N-way K-shot task (S, Q) from S 3:Compute e 0 x = G(x; 𝛾 0 , 𝛽 0 ) for all x ∈ S ∪ Q 4:Compute L 𝑐 for all x 𝑡𝑒𝑠𝑡 ∈ Q with 𝑓 𝑚 5:Reset h 0 = 0 6:for all 𝑡 = 1, • • • , T do 7:Compute S 𝑡 = {E 𝑡 , 𝛾 𝑡 -1 , 𝛽 𝑡 -1 } for all e 𝑡 -1x 8: Compute A 𝑡 = {𝛾 𝑡 , 𝛽 𝑡 } for S 𝑡 9: Compute e 𝑡 x = G(x; 𝛾 𝑡 , 𝛽 𝑡 ) for all x ∈ S ∪ Q 10:Compute 𝑟 𝑡 for all x 𝑡𝑒𝑠𝑡 ∈ Q with 𝑓 𝑚 EQUATIONwhere f 𝑜 𝜃 is a linear projection and 𝜏 denotes the sequence of dropout [45] and layer normalization [52] . Finally, E is calculated via the average pooling operation, denoted by AvgPool(•). Denote 𝜃 𝑔 as the set of all learnable parameters of the task state encoder.We choose 3 popular classifiers: ProtoNet [44] , DSN [42] , and MetaOptNet [23] . The corresponding models are named RMO-PN, RMO-DSN, and RMO-MON, respectively. Denote the predicted probability distribution of test sample (x 𝑞 , y 𝑞 ) as 𝑝 ( ŷ𝑞 = y 𝑞 | e x 𝑞 ; f 𝑚 ), where f 𝑚 denotes the classifier. We first define the classification loss as:EQUATIONThen, in each time step 𝑡, the log-probability is calculated as the reward:EQUATION)The goal is to maximize the accumulated rewards {𝑟 𝑡 } 𝑇 𝑡 =1 . Denote γ as the discount factor. We define the reinforcement loss as:EQUATIONwhere R 𝑡 = T -𝑡 𝑘=0 γ 𝑘 r 𝑡 +𝑘 denotes a discounted accumulated reward, π ∼ N (𝜇 𝑡 , 1). The total loss is defined as:EQUATIONwhere 𝜌 is a scalar hyper-parameter. The pseudocode of our approach is described in Algorithm 1.  
implementation details: 
 We utilize 4 commonly used backbones in few-shot learning: Conv-4 [21] , ResNet-12 [15] , ResNet-18 [15] , and WRN-28-10 [58] . The backbone takes a 3×84×84 image as input. For Conv-4, the Adam optimizer is utilized. For ResNets and WRN, the SGD optimizer is utilized with momentum of 0.9 and weight decay of 0.0005. We insert a FiLM layer after the last BN layer in the convolution blocks of the backbone. . First, we pre-train the backbone following the settings of the prior work [55] . Then, the whole model is meta-trained for 200 epochs, with each epoch including 100 randomly sampled tasks. We scale the learning rate of weights of the backbone by 0.1 in meta-training. In validation, we use 600 randomly sampled tasks to choose the best model. During evaluation, we report the classification accuracy and 95% interval confidence over 10000 randomly sampled tasks. Each task includes 15 query samples per class in all settings. We take the averaged results using 3 different random seeds. The learning rate is set to 0.002 and 0.001 for ConvNet and three ResNets, respectively. We scale the initial learning rate by 0.2 after every 30 epochs in metatraining. Our code is based on PyTorch [31] , and all experiments are conducted using an NVIDIA RTX A6000 GPU. More details are provided in Supplementary Material.Table 1 , Table 2 , and Table 3 list the results of our method on MiniImageNet, TieredImageNet, and CUB-200-2011, respectively. We mainly compare our approach with the previous state-of-the-art transductive few-shot learning approaches, including Curvature [13] , TEAM [36] , Oblique [35] , EPNet [41] , TIM [7] , ICI [50] , Lapla-cianShot [61] , and BD-CSPN [26] . It can be seen that our approach shows consistent improvements over other transductive few-shot learning approaches on all three datasets using the four backbones. For example, under 1-shot and 5-shot on MiniImageNet with ResNet-12, our approach shows on average +2.3%, +5.8%, and +4.2% performance gains comparing to Curvature [13] , CAN [18] , and MeTAL [5] , respectively. On TieredImageNet with ResNet-12, our approach shows on average +3.0%, +5.6%, and +6.4% performance gains comparing to the above three methods. On CUB-200-2011, our approach also shows on average +2.9% and +4.8% performance gains with TIM [7] and EPNet [41] , respectively. The promising results suggest that our approach makes it easier to reach a better local optima on the loss surface of the meta-training phase via exploring stable meta-optimization patterns, thus improving the discriminative ability of global embeddings.In this section, we first explore hyper-parameters' impacts on the proposed strategy. Then, we analyze the impacts of different modules on classification performance. Besides, we extend our strategy to the DenseNet backbone. Furthermore, we conduct experiments on the challenging cross-domain and generalized few-shot tasks. For simplicity, "PN", "DSN", and "MON" denote RMO-PN, RMO-DSN, and RMO-MON, respectively. Selection of hyper-parameters. We describe how we choose the value of hyper-parameter T , D, and γ in Supplementary Material.Impacts of different modules. We analyze the impact of each module, i.e., policy, memory, and task state encoder (denoted as "GRU", "DND", and "SATT", respectively). The experiments are conducted under 5-shot 5-way on MiniImageNet. Relative results are listed in Table 6 . To show the effects of differentiable RL and memory mechanism, we replace GRU and DND with two FC layers RMO-DSN (In.) 50.40±0.03 69.02±0.17 45.13 ±0.04 to learn the parameters 𝛾 and 𝛽. To show the effects of information adaptation from query set when constructing task state, we replace SATT with an averaging operation, i.e., obtaining the task state via averaging the mean vectors represented by each class as Oreshkin et al. [30] . It can be observed that the differentiable RL strategy and memory mechanism bring about +3% performance improvement, comparing to learning FiLM layer parameters with FC layers. Besides, the information adaptation from query set also shows about +2% performance improvement. Time complexity. The time complexity in each time step for SATT, GRU, and DND are 𝑂 ((𝑁 𝐾) 2 𝑑), 𝑂 (𝑑 2 ), and 𝑂 (D𝑑), respectively. The total time complexities of the three models are listed in Table 9 . Here we assume 𝑑 3 ≫ (𝑁 𝐾) 2 ≫ D ≫ 𝑑.Standard deviation of loss function. The standard deviation of the loss function can verify the stabilizing effects of our approach. We calculate the standard deviation of the loss function in the last 100 epochs. The experiments are conducted under 5-way 5-shot and 5-way 1-shot on MiniImageNet. Table 10 lists the results. It can be seen that our three models achieve lower standard deviations comparing to the three baseline models [23, 42, 44] .DenseNet backbone performance. Following the settings of Lichtenstein et al. [25] , we measure the performance of our approach with DenseNet [19] backbone. Specifically, we insert a FiLM layer after the last BN layer in each dense block of DenseNet-121. The backbone is first pre-trained following the protocol of prior work [49] . The experiments are conducted under 5-way 5-shot and 5-way 1-shot on MiniImageNet and TieredImageNet. Relative results are listed in Table 4 . Our approach outperforms previous stateof-the-art transductive methods by about +1.8%, including RAP-LaplacianShot [17] and TAFSSL [25] . This suggests our approach also contributes to dense connection-based backbones. More details are provided in Supplementary Material.Comparisons with typical task-adaptive methods. We compare the few-shot classification performance of our approach with two typical state-of-the-art approaches using FiLM layers: CNAPS [39] and Simple-CNAPS [6] . For a fair comparison, we utilize ResNet-18 pre-trained on ImageNet as backbone, and we insert a FiLM layer after the last BN layer in each residual block of ResNet-18. The model is meta-trained following the settings in Simple-CNAPS [6] .Since the works [6, 39] are inductive methods, we utilize only the support samples when calculating the task state (i.e., use all support samples as query instead of all query samples). Relative results are listed in Table 5 . Our method outperforms Simple-CNAPS by about +1% on average on MiniImageNet and TieredImageNet. This suggests our method also takes impacts with feature-rich backbones, comparing to directly learning parameters of FiLM layers. More details are provided in Supplementary Material. Cross-domain few-shot classification performance. Following the splits introduced by Chen et al. [8] , we meta-train the model on the entire MiniImageNet dataset (i.e., training set, validation set, and testing set), then meta-test the model on the test split of CUB-200-2011 dataset. Relative results are listed in Table 7 . We show obvious improvements in our approach under both 1-shot and 5-shot settings compared to other state-of-the-art cross-domain methods. This suggests that the differentiable RL strategy contributes to improving the discriminative ability of task-specific embeddings even when a domain gap exists.Generalized few-shot classification performance. Following the splits of Ye et al. [55] , we first train the model only on seen classes S of MiniImageNet dataset under 1-shot 5-way and 5-shot 5-way. Then we evaluate the model with three criteria: UNSEEN evaluates the 5-way classification accuracy only on unseen classes U; SEEN evaluates the 64-way classification accuracy only on seen classes S; COMBINED evaluates the 69-way mixed classification accuracy on both seen and unseen classes. We use the inductive versions of our method in the experiments. Relative results are listed in Table 8 . Our method shows about +4.3% and +6.9% performance gains against FEAT [55] and ProtoNet [44] respectively on the 3 evaluation criteria. This suggests that our approach also contributes to not forgetting previously learned class knowledge.Visualization. We depict the statistical distributions, represented as boxplots, of the parameters 𝛾 and 𝛽 in each time step 𝑡 across different meta-tasks in Figure 4 . We visualize the parameters 𝛾 and 𝛽 for the last convolution block of Conv-4 backbone. It can be observed that the parameters 𝛾 and 𝛽 have larger influence on the embedding procedure with longer time steps. Further, it can be seen that the generated parameter values vary among different tasks to a larger extent when the time step goes longer, especially at the last time step T . This suggests that the model learns to adapt to each meta-task generally across time steps via differentiable RL. Besides, we plot the curves of validation accuracy on MiniImageNet and TieredImageNet among our models and ProtoNet. The results are reported in Supplementary Material. 
conclusion: 
 This work proposes a generic differentiable RL-based meta optimization strategy for few-shot learning, aiming to explore stable metaoptimization patterns in meta-training. Due to the differentiability of the environment, our approach can avoid the flaws of black-box RL. This approach is a new way of introducing RL strategy into few-shot learning. The approach contains a policy equipped with the memory for learning generalizable optimizations of embedding procedure across unseen tasks. In addition, we propose a task state encoder to adapt information from the query set to the support set for fully exploring inner-task similarities. Our strategy can be easily applied to various backbones, including ConvNet, ResNet, WRN, and DenseNet. Our method also shows promising results on the challenging cross-domain and generalized few-shot classification tasks. Lastly, this work shows the huge benefits of exploring stable patterns in meta-learning, however it has limitations on model complexity. Future work includes studying the model efficiency and applying this approach to multiple downstream tasks.  
 
 Appendix: Submission ID: 956. 2024-04-11 19:58. Page 2 of 1-9.Submission ID: 956. 2024-04-11 19:58. Page 3 of 1-9.Submission ID: 956. 2024-04-11 19:58. Page 4 of 1-9.Submission ID: 956. 2024-04-11 19:58. Page 5 of 1-9.Submission ID: 956. 2024-04-11 19:58. Page 6 of 1-9.Submission ID: 956. 2024-04-11 19:58. Page 7 of 1-9.Submission ID: 956. 2024-04-11 19:58. Page 8 of 1-9.Submission ID: 956. 2024-04-11 19:58. Page 9 of 1-9. 


Decision: This work proposed a differentiable reinforcement learning framework for few-shot classification tasks. After the rebuttal period, this paper received consistent positive ratings. After checking the paper, reviews, and rebuttal document, I decide to accept this paper. 

However, as Reviewer Ju72 pointed out, this paper still has some small blemishes in expression. The authors need to fix this in the final version.

Reviews: review: paper_summary: paper_summary: This work proposed a differentiable reinforcement learning framework for few-shot classification tasks. The authors used a Gated Recurrent
Unit (GRU) as the policy module to control the parameters in FiLM layers. The policy module takes task embeddings as input. It also includes a memory module based on task embeddings to retreive the previous task that is most related. The embedding of a task is computed with a "self-attention" module that update embeddings of query samples by attending to support samples. Intensive experiments on different combinations of benchmark datasets and backbone architectures showed significant and consistent improvements over baselines. The authors also conducted complete ablation studies to show the effect brought by each module proposed.
strength_weakness: strength_weakness: + The paper is written in good quality, with proposed modules explained and illustrated clearly and experiments structured in an organized way.
+ The proposed method, in my opinion is novel. And the empirical results are impressive, with huge improvements over previous methods.
+ The ablation study is complete, providing convincing arguments of the effectiveness of eeach module proposed.
limitations: limitations: - It is good to also add a "baseline" row in Table. 6.
- Some combinations of previous methods were missing in the exp section. For example, the authors of RAP [17] reported RAP+LaplacianShot results on miniImageNet using ResNet-12 backbone, which showed a better 1-shot accuracy of 74.29±0.20%.
paper_summary: paper_summary: The paper aims to address the challenges of improving learning stability during meta training. The authors introduce a novel differentiable reinforcement learning strategy RMO to explore stable meta-optimization patterns to enhance the discriminative ability of task-adaptive embeddings. The authors claim that this approach can optimize the gradient flow in the meta-learning process with its differentiable reinforcement learning framework, and thus can make the few-shot learning models to be more robust and efficient.
strength_weakness: strength_weakness: The paper enhances few-shot learning with two key innovations. 
First, this paper introduces a new differentiable reinforcement learning framework that can stably manage meta-learners' optimization process. This framework can help maintain stable training across different tasks and address some common issues in traditional RL including the high variance and noisy gradients.
Second, this paper introduces a memory module to learn the continuity between different tasks. This approach is effective when dealing with learning process with limited samples while keeping consistency and stability.
The RMO method improves the model's discriminative performance, and the results of experiments show that RMO consistently outperforms existing state-of-the-art methods in standard benchmarks. Also, the experiments prove that the RMO's ability to dynamically adjust according to the specific needs of certain tasks along with the integrated memory module can enhance the model's performance and reliability, especially in real-world scenarios of limited data and varying tasks.
limitations: limitations: This paper could benefit from several improvements:
First, the article should introduce the principles and characteristics of the FiLM model in the RELATED WORK section before its usage. This would enhance the readability of the paper.
Second, although the authors present the results on multiple datasets, they may consider the balance of the data, which would help to explain the model's robustness on both balanced and imbalanced datasets and further prove the RMO's improvement on stability.
Additionally, integrating various types of attention mechanisms may further enhance the RMO's ability to adapt dynamically to complex tasks, especially in multi-category datasets.
Finally, the Figure 4 needs adjustments for better clarity. There is a wide blank while the key information is folded.
paper_summary: paper_summary: This paper presents a differentiable reinforcement learning (RL) strategy for few-shot classification. The aim is to explore stable meta-optimization patterns in meta-training by learning generalizable optimizations for producing task-adaptive embeddings. The proposed strategy models the embedding procedure of feature transformation layers in the meta-learner to optimize the gradient flow implicitly.  The approach is demonstrated to be effective across various backbones and achieves promising results against state-of-the-art transductive methods in few-shot classification.
strength_weakness: strength_weakness: 1.The proposed method is solid.

2.The approach is generic and can be easily extended to various backbones, including ConvNet, ResNet, WRN, and DenseNet.

3.The proposed method has been evaluated on three popular datasets and has shown consistent improvement in performance across different backbones.
limitations: limitations: 1.To improve the evaluation, the authors should consider including more recent works. The most recent baseline cited is from 2021, which may not adequately reflect the current advancements in the field.

2.The related work section reviews current issues well but lacks details on how this paper addresses them. Please clearly outline the novel contributions and improvements made.
paper_summary: paper_summary: This paper combines reinforcement learning into meta-optimization in few shot learning.  By designing the reward, state and action in reinforcement learning, the proposed approach can generate the parameters \gama and \beta in the BN layer. It is an interesting idea and experiments prove the effectiveness of this approach.
strength_weakness: strength_weakness: 1. The work for this kind of combination is interesting,  that adopts generic differentiable RL-based few-shot learning strategy to explore stable meta-optimization patterns for meta-training. 
2. The writting is fine for reading.
3. Experiments are ok for me, especially the results on the challenging cross-domain and generalized few-shot classification tasks.
limitations: limitations: 1. The gradient updating step T is set as five, which is not a smaller number in many meta-learning approaches. 
2. The identification of some variables can be clearer, while some variables do not have explicit meanings.