Miracle-dz/UniFAR-FTUs huggingface dataset:
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning

The lottery ticket hypothesis states that a highly sparsified sub-network can be trained in isolation, given the appropriate weight initialization. This paper extends that hypothesis from one-shot task leaning, and demonstrates for the first time that such extremely compact and independently trainable sub-networks can be also identified in the lifelong learning scenario, which we call lifelong tickets. We show that the resulting lifelong ticket can further be leveraged to improve the performance of learning over continual tasks. However, it is highly non-trivial to conduct network pruning in the lifelong setting. Two critical roadblocks arise: i) As many tasks now arrive sequentially, finding tickets in a greedy weight pruning fashion will inevitably suffer from the intrinsic bias, that the earlier emerging tasks impact more; ii) As lifelong learning is consistently challenged by catastrophic forgetting, the compact network capacity of tickets might amplify the risk of forgetting. In view of those, we introduce two pruning options, e.g., top-down and bottom-up, for finding lifelong tickets. Compared to the top-down pruning that extends vanilla (iterative) pruning over sequential tasks, we show that the bottomup one, which can dynamically shrink and (re-)expand model capacity, effectively avoids the undesirable excessive pruning in the early stage. We additionally introduce lottery teaching that further overcomes forgetting via knowledge distillation aided by external unlabeled data. Unifying those ingredients, we demonstrate the existence of very competitive lifelong tickets, e.g., achieving 3− 8% of the dense model size with even higher accuracy, compared to strong class-incremental learning baselines on CIFAR-10/CIFAR-100/Tiny-ImageNet datasets.
Exploring Balanced Feature Spaces for Representation Learning

Existing self-supervised learning (SSL) methods are mostly applied for training representation models from artiﬁcially balanced datasets ( e.g . ImageNet). It is unclear how well they will perform in the practical scenarios where datasets are often imbalanced w.r. t. the classes. Motivated by this question, we conduct a series of studies on the performance of self-supervised contrastive learning and supervised learning methods over multiple datasets where training instance distributions vary from a balanced one to a long-tailed one. Our ﬁndings are quite intriguing. Different from supervised methods with large performance drop, the self-supervised contrastive learning methods perform stably well even when the datasets are heavily imbalanced. This motivates us to explore the balanced feature spaces learned by contrastive learning, where the feature representations present similar linear separability w.r. t. all the classes. Our further experiments reveal that a representation model generating a balanced feature space can generalize better than that yielding an imbalanced one across multiple settings. Inspired by these insights, we develop a novel representation learning method, called k -positive contrastive learning. It effectively combines strengths of the supervised method and the contrastive learning method to learn representations that are both discriminative and balanced. Extensive experiments demonstrate its superiority on multiple recognition tasks, including both long-tailed ones and normal balanced ones. Code is available at https://github.com/bingykang/BalFeat .
Genetic Soft Updates for Policy Evolution in Deep Reinforcement Learning

The combination of Evolutionary Algorithms (EAs) and Deep Reinforcement Learning (DRL) has been recently proposed to merge the beneﬁts of both solutions. Existing mixed approaches, however, have been successfully applied only to actor-critic methods and present signiﬁcant overhead. We address these issues by introducing a novel mixed framework that exploits a periodical genetic evaluation to soft update the weights of a DRL agent. The resulting approach is applicable with any DRL method and, in a worst-case scenario, it does not exhibit detrimental behaviours. Experiments in robotic applications and continuous control benchmarks demonstrate the versatility of our approach that signiﬁcantly outperforms prior DRL, EAs, and mixed approaches. Finally, we employ formal veriﬁcation to conﬁrm the policy improvement, mitigating the inefﬁcient exploration and hyper-parameter sensitivity of DRL
Influence of humectants on the drying kinetics, water mobility, and moisture sorption isotherm of osmosed air-dried papaya

ABSTRACT Papaya slices were sequentially soaked in 40, 50, and 60° Brix sucrose solutions. There was a higher water loss and solids gain when 60° Brix sucrose solution was partially substituted with 10% (w/v) sorbitol, 10% (v/v) glycerol, or invert sugar. The modified Henderson and Pabis model provided the best fit to experimental drying kinetics. Moisture sorption isotherm of all dried papaya, at 30 ± 2°C, was a type III with different curvature, revealing different sugar-binding ability towards water molecules. The Henderson model showed suitable fit to experimental desorption isotherm data. NMR water mobility revealed the better ability of glycerol to bind water than sucrose.
NBDT: Neural-Backed Decision Tree

Machine learning applications such as finance and medicine demand accurate and justifiable predictions, barring most deep learning methods from use. In response, previous work combines decision trees with deep learning, yielding models that (1) sacrifice interpretability for accuracy or (2) sacrifice accuracy for interpretability. We forgo this dilemma by jointly improving accuracy and interpretability using Neural-Backed Decision Trees (NBDTs). NBDTs replace a neural network’s final linear layer with a differentiable sequence of decisions and a surrogate loss. This forces the model to learn high-level concepts and lessens reliance on highlyuncertain decisions, yielding (1) accuracy: NBDTs match or outperform modern neural networks on CIFAR, ImageNet and better generalize to unseen classes by up to 16%. Furthermore, our surrogate loss improves the original model’s accuracy by up to 2%. NBDTs also afford (2) interpretability: improving human trust by clearly identifying model mistakes and assisting in dataset debugging. Code and pretrained NBDTs are at github.com/alvinwan/neural-backed-decision-trees.
Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning

Understanding human behavior from observed data is critical for transparency and accountability in decision-making. Consider real-world settings such as healthcare, in which modeling a decision-maker's policy is challenging -- with no access to underlying states, no knowledge of environment dynamics, and no allowance for live experimentation. We desire learning a data-driven representation of decision-making behavior that (1) inheres transparency by design, (2) accommodates partial observability, and (3) operates completely offline. To satisfy these key criteria, we propose a novel model-based Bayesian method for interpretable policy learning ("Interpole") that jointly estimates an agent's (possibly biased) belief-update process together with their (possibly suboptimal) belief-action mapping. Through experiments on both simulated and real-world data for the problem of Alzheimer's disease diagnosis, we illustrate the potential of our approach as an investigative device for auditing, quantifying, and understanding human decision-making behavior.
Drug Disease Relation Extraction from Biomedical Literature Using NLP and Machine Learning

Extracting the relations between medical concepts is very valuable in the medical domain. Scientists need to extract relevant information and semantic relations between medical concepts, including protein and protein, gene and protein, drug and drug, and drug and disease. These relations can be extracted from biomedical literature available on various databases. This study examines the extraction of semantic relations that can occur between diseases and drugs. Findings will help specialists make good decisions when administering a medication to a patient and will allow them to continuously be up to date in their field. The objective of this work is to identify different features related to drugs and diseases from medical texts by applying Natural Language Processing (NLP) techniques and UMLS ontology. The Support Vector Machine classifier uses these features to extract valuable semantic relationships among text entities. The contributing factor of this research is the combination of the strength of a suggested NLP technique, which takes advantage of UMLS ontology and enables the extraction of correct and adequate features (frequency features, lexical features, morphological features, syntactic features, and semantic features), and Support Vector Machines with polynomial kernel function. These features are manipulated to pinpoint the relations between drug and disease. The proposed approach was evaluated using a standard corpus extracted from MEDLINE. The finding considerably improves the performance and outperforms similar works, especially the f-score for the most important relation “cure,” which is equal to 98.19%. The accuracy percentage is better than those in all the existing works for all the relations.
Knowledge distillation via softmax regression representation learning

This paper addresses the problem of model compression via knowledge distillation. We advocate for a method that optimizes the output feature of the penulti-mate layer of the student network and hence is directly related to representation learning. To this end, we ﬁrstly propose a direct feature matching approach which focuses on optimizing the student’s penultimate layer only. Secondly and more importantly , because feature matching does not take into account the classiﬁcation problem at hand, we propose a second approach that decouples representation learning and classiﬁcation and utilizes the teacher’s pre-trained classiﬁer to train the student’s penultimate layer feature. In particular, for the same input image, we wish the teacher’s and student’s feature to produce the same output when passed through the teacher’s classiﬁer, which is achieved with a simple L 2 loss. Our method is extremely simple to implement and straightforward to train and is shown to consistently outperform previous state-of-the-art methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains. The code is available at https://github.com/jingyang2017/KD_SRRL .
FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning

Federated learning aims to collaboratively train a strong global model by accessing users' locally trained models but not their own data. A crucial step is therefore to aggregate local models into a global model, which has been shown challenging when users have non-i.i. d. data. In this paper, we propose a novel aggregation algorithm named FedBE, which takes a Bayesian inference perspective by sampling higher-quality global models and combining them via Bayesian model Ensemble, leading to much robust aggregation. We show that an effective model distribution can be constructed by simply fitting a Gaussian or Dirichlet distribution to the local models. Our empirical studies validate FedBE's superior performance, especially when users' data are not i.i. d. and when the neural networks go deeper. Moreover, FedBE is compatible with recent efforts in regularizing users' model training, making it an easily applicable module: you only need to replace the aggregation method but leave other parts of your federated learning algorithm intact. Our code is publicly available at https://github.com/hongyouc/FedBE.
ResNet After All? Neural ODEs and Their Numerical Solution

A key appeal of the recently proposed Neural Ordinary Differential Equation (ODE) framework is that it seems to provide a continuous-time extension of discrete residual neural networks. As we show herein, though, trained Neural ODE models actually depend on the specific numerical method used during training. If the trained model is supposed to be a flow generated from an ODE, it should be possible to choose another numerical solver with equal or smaller numerical error without loss of performance. We observe that if training relies on a solver with overly coarse discretization, then testing with another solver of equal or smaller numerical error results in a sharp drop in accuracy. In such cases, the combination of vector field and numerical method cannot be interpreted as a flow generated from an ODE, which arguably poses a fatal breakdown of the Neural ODE concept. We observe, however, that there exists a critical step size beyond which the training yields a valid ODE vector field. We propose a method that monitors the behavior of the ODE solver during training to adapt its step size, aiming to ensure a valid ODE without unnecessarily increasing computational cost. We verify this adaptation algorithm on a common bench mark dataset as well as a synthetic dataset.
Multiplicative Filter Networks

Although deep networks are typically used to approximate functions over high dimensional inputs, recent work has increased interest in neural networks as function approximators for low-dimensional-but-complex functions, such as representing images as a function of pixel coordinates, solving differential equations, or representing signed distance functions or neural radiance fields. Key to these recent successes has been the use of new elements such as sinusoidal nonlinearities or Fourier features in positional encodings, which vastly outperform simple ReLU networks. In this paper, we propose and empirically demonstrate that an arguably simpler class of function approximators can work just as well for such problems: multiplicative filter networks. In these networks, we avoid traditional compositional depth altogether, and simply multiply together (linear functions of) sinusoidal or Gabor wavelet functions applied to the input. This representation has the notable advantage that the entire function can simply be viewed as a linear function approximator over an exponential number of Fourier or Gabor basis functions, respectively. Despite this simplicity, when compared to recent approaches that use Fourier features with ReLU networks or sinusoidal activation networks, we show that these multiplicative filter networks largely outperform or match the performance of these approaches on the domains highlighted in these past works.
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation

Much recent effort has been invested in non-autoregressive neural machine translation, which appears to be an efficient alternative to state-of-the-art autoregressive machine translation on modern GPUs. In contrast to the latter, where generation is sequential, the former allows generation to be parallelized across target token positions. Some of the latest non-autoregressive models have achieved impressive translation quality-speed tradeoffs compared to autoregressive baselines. In this work, we reexamine this tradeoff and argue that autoregressive baselines can be substantially sped up without loss in accuracy. Specifically, we study autoregressive models with encoders and decoders of varied depths. Our extensive experiments show that given a sufficiently deep encoder, a single-layer autoregressive decoder can substantially outperform strong non-autoregressive models with comparable inference speed. We show that the speed disadvantage for autoregressive baselines compared to non-autoregressive methods has been overestimated in three aspects: suboptimal layer allocation, insufficient speed measurement, and lack of knowledge distillation. Our results establish a new protocol for future research toward fast, accurate machine translation. Our code is available at https://github.com/jungokasai/deep-shallow.
Off-Policy Multi-Agent Decomposed Policy Gradients

Multi-agent policy gradient (MAPG) methods recently witness vigorous progress. However, there is a significant performance discrepancy between MAPG methods and state-of-the-art multi-agent value-based approaches. In this paper, we investigate causes that hinder the performance of MAPG algorithms and present a multi-agent decomposed policy gradient method (DOP). This method introduces the idea of value function decomposition into the multi-agent actor-critic framework. Based on this idea, DOP supports efficient off-policy learning and addresses the issue of centralized-decentralized mismatch and credit assignment in both discrete and continuous action spaces. We formally show that DOP critics have sufficient representational capability to guarantee convergence. In addition, empirical evaluations on the StarCraft II micromanagement benchmark and multi-agent particle environments demonstrate that DOP significantly outperforms both state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms. Demonstrative videos are available at this https URL.
PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences

Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension. Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner. Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.
Antecedents of science anxiety in elementary school

Abstract Based on the control-value theory of achievement emotions, the study examines elementary school students’ science performance and control-value beliefs (self-concept, intrinsic and achievement value) as antecedents of their cognitive (worry) and affective (nervousness) science anxiety. Performance in the science topic “floating and sinking” was assessed and self-reported data on control-value beliefs and worry and nervousness about science were collected from 370 students at three time points during third grade. Results from structural equation modeling support the control-value theory and show that self-concept completely mediated the negative effect of prior science performance on both anxiety components. By contrast, value beliefs did not function as mediators and only achievement value positively predicted nervousness and worry, whereas no effect of intrinsic value was observed. Results are invariant across gender and indicate stereotypical mean differences in science anxiety components that were not due to differences in the predictor variables.
Sustainable rural tourism: linking residents’ environmentally responsible behaviour to tourists’ green consumption

ABSTRACT Based on the Stimulus-Organism-Response framework, this research sought to examine tourists’ environmental concern and destination identification as mediating variables between residents’ environmentally responsible behaviour and tourists’ green consumption in Chinese rural tourism. A total of 275 participants were recruited, and structural equation modelling was conducted to test hypotheses. Findings indicate residents’ environmentally responsible behaviour has a positive relationship with tourists’ green consumption; environmental concern and destination identification mediated this relationship. Moreover, tourists’ destination identification was found to impact their environmental concerns significantly. The present study contributes to promoting the sustainable development of rural tourism.
Chinese Perceptions of Rio's destination image before and after the 2016 Olympics

ABSTRACT This study investigates Chinese perceptions of Rio’s destination image before and after the 2016 Olympics. Samples of 573 and 564 residents from Shanghai were surveyed before and after the event. The Olympics raised awareness of Rio and as an Olympic host city. Post-games perceptions of affective image and tourist attractions were higher for people aware that Rio had hosted the Olympics. However, pre-games perceptions of tourism infrastructure and service in Rio were lower for participants aware that Rio would host the Olympics, and that the post-games perceptions of tourism infrastructure and service were lower than pre-games perceptions.
Do hedonic and utilitarian values increase pro-environmental behavior and support for festivals?

ABSTRACT This study integrated a theory of consumption values and place attachment theory to hypothesize the relationships among visitors’ consumption values such as hedonic and utilitarian values and their impact on place attachment, attitude, pro-environmental behavior, and support for tourism. The results demonstrate that hedonic and utilitarian values play an important role in predicting place attachment and attitude. Place attachment is found to be significant in measuring attendees’ attitude toward pro-environmental behavior and support for tourism. This study supports the theory of consumption values such that festival managers should consider hedonic and utilitarian values of the attendees’ attitude.
Examining destinations’ personality and brand equity through the lens of expats: moderating role of expat’s cultural intelligence

ABSTRACT The role of expat’s community has emerged as a cornerstone for branding tourism destinations. This study focused on 307 expats (across Korea and Turkey) and investigated the effects of perceived destination personality (DP) on destination brand equity (DBE), under the moderating influence of expat’s cultural intelligence (ECI). The findings empirically confirmed the significant positive effect of DP on DBE, however, the cross-country differences in DBE were significantly influenced by ECI. The study findings offer strategic directions for destination marketers to re-assess the vital role of global expats in boosting DP and DBE, especially when ECI remains superior over general tourists.
Clovis Technology is not Unique to Clovis

ABSTRACT We previously showed that stone-tool technological attributes thought to be unique to the Clovis period were present in a radiocarbon and OSL dated middle Holocene-age stratum at Goodson Shelter, Oklahoma (Eren et al. 2018a. “Is Clovis Technology Unique to Clovis? ” PaleoAmerica 4:202–228). Consequently, we argued that technological attributes alone should not be used to assign assemblages to Clovis times. Huckell, Haynes, and Holliday (2019. “Comments on the Lithic Technology and Geochronology of the Goodson Rock Shelter.” PaleoAmerica 6:131–134) proposed two alternative hypotheses: that material we identified as Clovis-like was not, or that it was Clovis but had been mixed with younger deposits. They called for more information on the Clovis-like assemblage at Goodson, and additional dating of the site's lowest deposits. We provide that information, which confirms that stone-tool technologies ostensibly unique to Clovis were indeed in use in the middle Holocene.