Kati8/ICLR_2023_2025 huggingface dataset:
authors: SangHwa Hong
authors: Bongseok Kim
authors: Min Kyu Sim
reviews: 
Comments: In summary, this paper introduces a new regularization method called Periodic Regularization, which involves adjusting hyperparameters periodically during training. When combined with Reinforcement Learning and other techniques, the authors show experimental results in improving performance, particularly in tasks like Facial Expression Recognition. The main goal is to advance research in regularization techniques and deep learning.
Strengths: The idea of periodic regularization is interesting and could potentially benefit deep learning optimization. It can also generalize by adopting existing and future regularization methods into its pipeline, and reducing the manual effort in applying regularization methods in real-world applications.

The paper is well-structured and the interpretation is clear.
Weaknesses: Methodology: 
There has been a lot of prior work on meta-learning, especially meta-RL. Given that the regularization parameters (e.g., dropout rate) are usually seen as hyperparameters, the authors should explain the connection between the proposed method and existing meta-learning methods such as MAML, meta-gradients, etc.
In addition, there are also variants of regularization methods that are trained along with the network, e.g., Adaptive Dropout. The authors should mention the literature and explain the differences and novelty.

Experiments:
The main issue with the paper is in experiments. First, the method is only tested on 3 small datasets with 1 ResNet each. It's hard to demonstrate the usability of the proposed method in today's systems. The authors should include more architectures (EfficientNet, ViT, etc.) and experiment on a larger dataset (ImageNet). 
Secondly, the reported results are not convincing and actually contradict existing work. For example, for ResNet34 on CIFAR-100, one should easily get >.75 accuracy with open-source implementations (ref: https://huggingface.co/edadaltocg/resnet34_cifar100,  https://github.com/weiaicunzai/pytorch-cifar100). The reported scores are way below the baseline, so it is hard to say if the proposed method actually improves the performance. 
Finally, the authors should compare to other methods such as MAML, adaptive dropout, etc.
Recommendation: 3: reject, not good enough
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 2 fair
reviews: 
Comments: The paper introduces Periodic Regularization, a technique that incorporates periodicity into the dynamic adjustment of regularization hyperparameters during training. By framing the hyperparameter tuning as a reinforcement learning or transfer learning problem, this approach allows for the application of techniques such as RL and fine-tuning to enhance performance. Through experiments, the paper shows that combining Periodic Regularization with RL consistently improves accuracy across a wide range of datasets, sometimes surpassing the performance of conventional regularization methods.
Strengths: The paper considers a crucial problem in deep learning to dynamically adjust hyperparameters over periods during training.
Weaknesses: The paper has many typos, format issues and grammatical mistakes, so the quality is quite poor. For example, 1) there is no space between a word and a reference in many sentences, 2) As *an* representative example..., 3) there is ?? in the caption of Fig. 3.

All figures are low-resolution and hard to read.
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: The paper introduces Periodic Regularization, a novel regularization technique that incorporates periodicity into the dynamic hyperparameter tuning. The paper suggests combining Periodic Regularization with other learning techniques such as Reinforcement Learning (RL) and Transfer Learning to further improve model performance. The authors demonstrate the effectiveness of Periodic Regularization in addressing the overfitting problem in Facial Expression Recognition (FER) tasks, where traditional regularization techniques have proven ineffective. The authors argue that Periodic Regularization can enhance model performance across different datasets and automatically adjust hyperparameters to prevent overfitting.
Strengths: + This paper shows that periodic regularization can improve models using different learning frameworks such as reinforcement learning and transfer learning.
+ The paper experiments show that Periodic Regularization can effectively solve the over-fitting problem in the FER task.
+ Periodic regularization can automatically adjust hyperparameters.
+ The idea is great and works well in practice.
Weaknesses: + The motivation is unclear. Heterogeneity is not well defined. It is confusing that we should "introduce heterogeneity in the hidden layer to enhance the adaptability and generalization of the model". Then, the author only explains how PR is designed, but ignores why it is designed this way.
+ Writing needs improvement. Please do not abuse table/picture hyperlinks in your paper.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 2 fair