Reviewerly/exHarmony huggingface dataset:
Decoupled Knowledge Distillation

State-of-the-art distillation methods are mainly based on distilling deep features from intermediate layers, while the significance of logit distillation is greatly overlooked. To provide a novel viewpoint to study logit distillation, we re-formulate the classical KD loss into two parts, i.e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD). We empirically investigate and prove the effects of the two parts: TCKD transfers knowledge concerning the “difficulty” of training samples, while NCKD is the prominent reason why logit distillation works. More importantly, we reveal that the classical KD loss is a coupled formulation, which (1) suppresses the effectiveness of NCKD and (2) limits the flexibility to balance these two parts. To address these issues, we present Decoupled Knowledge Distillation (DKD), enabling TCKD and NCKD to play their roles more efficiently and flexibly. Compared with complex feature-based methods, our DKD achieves comparable or even better results and has better training efficiency on CIFAR-100, ImageNet, and MS-COCO datasets for image classification and object detection tasks. This paper proves the great potential of logit distillation, and we hope it will be helpful for future research. The code is available at https://github.com/megviiresearch/mdistiller.
Robust Unscented Kalman Filter for Power System Dynamic State Estimation With Unknown Noise Statistics

Due to the communication channel noises, GPS synchronization process, changing environment temperature and different operating conditions of the system, the statistics of the system process and measurement noises may be unknown and they may not follow Gaussian distributions. As a result, the traditional Kalman filter-based dynamic state estimators may provide strongly biased state estimates. To address these issues, this paper develops a robust generalized maximum-likelihood unscented Kalman filter (GM-UKF). The statistical linearization approach is presented to derive a compact batch-mode regression form by processing the predicted state vector and the received measurements simultaneously. This regression form enhances the data redundancy and allows us to detect bad phasor measurement unit measurements and incorrect state predictions, and filter out unknown Gaussian and non-Gaussian noises through the generalized maximum likelihood-estimator. The latter minimizes a convex Huber function with weights calculated via the projection statistics (PS). Particularly, the PS is applied to a proposed 2-dimensional matrix that consists of temporally correlated innovation vectors and predicted states. Finally, the total influence function is used to derive the error covariance matrix of the GMUKF state estimates, yielding the robust state prediction at the next time instant. Extensive simulations carried out on the IEEE 39-bus test system demonstrate the effectiveness and robustness of the proposed method.
Dual watermarking framework for privacy protection and content authentication of multimedia

In the current multimedia networked infrastructure, privacy breaches due to cyber-attacks result in huge economic losses. Despite these threats there is ever increasing demand to share data over various insecure networks for accomplishment of numerous tasks. In such a scenario there is a greater need to develop new algorithms for strengthening the existing cybersecurity frameworks, ensure security, privacy, copyright protection and authentication of data. In this paper a new technique for copyright protection, data security and content authentication of multimedia images is presented. The copyright protection of the media is taken care of by embedding a robust watermark using an efficient inter-block coefficient differencing algorithm and is proposed as Scheme I. Scheme II has been utilized to ensure both copyright protection, and content authentication. The authentication of the content has been ensured by embedding a fragile watermark in spatial domain while as copyright protection has been taken care of utilizing a robust watermark. In order to thwart an adversary and ensure that it has no access to actual embedded data, we make use of a novel encryption algorithm in conjunction with Arnold transform to encrypt data prior to its embedding. The experimental results reveal that the proposed framework offers high degree of robustness against single/dual/triple attacks; with Normalized Correlation (NCC), more than or equal to 0.95. Besides, the fragile watermark embedding makes the system capable of tamper detection and localization with average BER more than 45% for all signal processing/geometric attacks. The average PSNR achieved for both schemes is greater than 41 dB. A comparison of the proposed framework with various state-of-the-art techniques demonstrate its effectiveness and superiority.
Combination of Support Vector Machine and K-Fold cross validation to predict compressive strength of concrete in marine environment

Support Vector Machine (SVM) model optimized by K-Fold cross validation was built to predict and evaluate the degradation of concrete strength in a complicated marine environment. Meanwhile, Artificial Neural Network (ANN) and Decision Tree (DT) were built to compare the prediction precision with SVM model. The result shows SVM model has the best prediction performance. For the original SVM model, there is a problem that input parameters of SVM model are incomplete. Based on the analysis of an amount of research papers, number of input parameters increases from original 8 to 19. After the complementation of input parameters, the prediction precision of SVM model gets a good promotion. The average relative error declines from 34.8% to 27.6%, and the median relative error is reduced from 24.70% to 20.8%. The number of data with relative errors less than 40% has reached 396, almost 80% of 500 testing data.
PARTIES

Multi-tenancy in modern datacenters is currently limited to a single latency-critical, interactive service, running alongside one or more low-priority, best-effort jobs. This limits the efficiency gains from multi-tenancy, especially as an increasing number of cloud applications are shifting from batch jobs to services with strict latency requirements. We present PARTIES, a QoS-aware resource manager that enables an arbitrary number of interactive, latency-critical services to share a physical node without QoS violations. PARTIES leverages a set of hardware and software resource partitioning mechanisms to adjust allocations dynamically at runtime, in a way that meets the QoS requirements of each co-scheduled workload, and maximizes throughput for the machine. We evaluate PARTIES on state-of-the-art server platforms across a set of diverse interactive services. Our results show that PARTIES improves throughput under QoS by 61% on average, compared to existing resource managers, and that the rate of improvement increases with the number of co-scheduled applications per physical host.
Fuzzy commonsense reasoning for multimodal sentiment analysis

The majority of user-generated content posted online is in the form of text, images and videos but also physiological signals in games. AffectiveSpace is a vector space of affective commonsense available for English text but not for other languages nor other modalities such as electrocardiogram signals. We overcome this limitation by using deep learning to extract features from each modality and then projecting them to a common AffectiveSpace that has been clustered into different emotions. Because, in the real world, individuals tend to have partial or mixed sentiments about an opinion target, we use a fuzzy logic classifier to predict the degree of a particular emotion in AffectiveSpace. The combined model of deep convolutional neural networks and fuzzy logic is termed Convolutional Fuzzy Sentiment Classifier. Lastly, because the computational complexity of a fuzzy classifier is exponential with respect to the number of features, we project features to a four dimensional emotion space in order to speed up the classification performance.
A Dilated CNN Model for Image Classification

The dilated convolution algorithm, which is widely used for image segmentation, is applied in the image classification field in this paper.In many traditional image classification algorithms, convolution neural network (CNN) plays an important role.However, the classical CNN has the problem of consuming too much computing resources.To solve this problem, first, this paper proposed a dilated CNN model which is built through replacing the convolution kernels of traditional CNN by the dilated convolution kernels, and then, the dilated CNN model is tested on the Mnist handwritten digital recognition data set.Second, to solve the detail loss problem in the dilated CNN model, the hybrid dilated CNN (HDC) is built by stacking dilated convolution kernels with different dilation rates, and then the HDC model is tested on the wide-band remote sensing image data set of earth's terrain.The results show that under the same environment, compared with the traditional CNN model, the dilated CNN model reduces the training time by 12.99% and improves the training accuracy by 2.86% averagely, compared with the dilated CNN model, the HDC model reduces the training time by 2.02% and improves the training and testing accuracy by 14.15% and 15.35% averagely.Therefore, the dilated CNN and HDC model proposed in this paper can significantly improve the image classification performance.
Temporal Cycle-Consistency Learning

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle-consistency (TCC), a differentiable cycle-consistency loss that can be used to find correspondences across time in multiple videos. The resulting per-frame embeddings can be used to align videos by simply matching frames using nearest-neighbors in the learned embedding space. To evaluate the power of the embeddings, we densely label the Pouring and Penn Action video datasets for action phases. We show that (i) the learned embeddings enable few-shot classification of these action phases, significantly reducing the supervised training requirements; and (ii) TCC is complementary to other methods of self-supervised learning in videos, such as Shuffle and Learn and Time-Contrastive Networks. The embeddings are also used for a number of applications based on alignment (dense temporal correspondence) between video pairs, including transfer of metadata of synchronized modalities between videos (sounds, temporal semantic labels), synchronized playback of multiple videos, and anomaly detection. Project webpage: https://sites.google.com/view/temporal-cycle-consistency .
Enforcing Analytic Constraints in Neural Networks Emulating Physical Systems

Neural networks can emulate nonlinear physical systems with high accuracy, yet they may produce physically inconsistent results when violating fundamental constraints. Here, we introduce a systematic way of enforcing nonlinear analytic constraints in neural networks via constraints in the architecture or the loss function. Applied to convective processes for climate modeling, architectural constraints enforce conservation laws to within machine precision without degrading performance. Enforcing constraints also reduces errors in the subsets of the outputs most impacted by the constraints.Received 17 September 2019Revised 9 November 2020Accepted 1 February 2021DOI:https://doi.org/10.1103/PhysRevLett.126.098302© 2021 American Physical SocietyPhysics Subject Headings (PhySH)Research AreasAtmospheric fluid dynamicsAtmospheric scienceClimate researchConvectionGeophysical fluid dynamicsOptimization problemsStratified geophysical flowsThermodynamicsTurbulencePhysical SystemsArtificial neural networksNetworksInterdisciplinary PhysicsNonlinear DynamicsFluid Dynamics
TASO

Existing deep neural network (DNN) frameworks optimize the computation graph of a DNN by applying graph transformations manually designed by human experts. This approach misses possible graph optimizations and is difficult to scale, as new DNN operators are introduced on a regular basis.
Prism

The concept of a blockchain was invented by Satoshi Nakamoto to maintain a distributed ledger. In addition to its security, important performance measures of a blockchain protocol are its transaction throughput and confirmation latency. In a decentralized setting, these measures are limited by two underlying physical network attributes: communication capacity and speed-of-light propagation delay. In this work we introduce Prism, a new proof-of-work blockchain protocol, which can achieve 1) security against up to 50% adversarial hashing power; 2) optimal throughput up to the capacity C of the network; 3) confirmation latency for honest transactions proportional to the propagation delay D, with confirmation error probability exponentially small in the bandwidth-delay product CD; 4) eventual total ordering of all transactions. Our approach to the design of this protocol is based on deconstructing Nakamoto's blockchain into its basic functionalities and systematically scaling up these functionalities to approach their physical limits.
Self-Ensembling With GAN-Based Data Augmentation for Domain Adaptation in Semantic Segmentation

Deep learning-based semantic segmentation methods have an intrinsic limitation that training a model requires a large amount of data with pixel-level annotations. To address this challenging issue, many researchers give attention to unsupervised domain adaptation for semantic segmentation. Unsupervised domain adaptation seeks to adapt the model trained on the source domain to the target domain. In this paper, we introduce a self-ensembling technique, one of the successful methods for domain adaptation in classification. However, applying self-ensembling to semantic segmentation is very difficult because heavily-tuned manual data augmentation used in self-ensembling is not useful to reduce the large domain gap in the semantic segmentation. To overcome this limitation, we propose a novel framework consisting of two components, which are complementary to each other. First, we present a data augmentation method based on Generative Adversarial Networks (GANs), which is computationally efficient and effective to facilitate domain alignment. Given those augmented images, we apply self-ensembling to enhance the performance of the segmentation network on the target domain. The proposed method outperforms state-of-the-art semantic segmentation methods on unsupervised domain adaptation benchmarks.
Joint Monocular 3D Vehicle Detection and Tracking

Vehicle 3D extents and trajectories are critical cues for predicting the future location of vehicles and planning future agent ego-motion based on those predictions. In this paper, we propose a novel online framework for 3D vehicle detection and tracking from monocular videos. The framework can not only associate detections of vehicles in motion over time, but also estimate their complete 3D bounding box information from a sequence of 2D images captured on a moving platform. Our method leverages 3D box depth-ordering matching for robust instance association and utilizes 3D trajectory prediction for re-identification of occluded vehicles. We also design a motion learning module based on an LSTM for more accurate long-term motion extrapolation. Our experiments on simulation, KITTI, and Argoverse datasets show that our 3D tracking pipeline offers robust data association and tracking. On Argoverse, our image-based method is significantly better for tracking 3D vehicles within 30 meters than the LiDAR-centric baseline methods.
Boosting salp swarm algorithm by sine cosine algorithm and disrupt operator for feature selection

Features Selection (FS) plays an important role in enhancing the performance of machine learning techniques in terms of accuracy and response time. As FS is known to be an NP-hard problem, the aim of this paper is to introduce basically a new variant of Salp Swarm Optimizer (SSA) for FS (called ISSAFD (Improved Followers of Salp swarm Algorithm using Sine Cosine algorithm and Disrupt Operator), that updates the position of followers (F) in SSA using sinusoidal mathematical functions that were inspired from the Sine Cosine Algorithm (SCA). This enhancement helps to improve the exploration phase and to avoid stagnation in a local area. Moreover, the Disruption Operator (Dop) is applied for all solutions, in order to enhance the population diversity and to maintain the balance between exploration and exploitation processes. Two other variants of SSA are developed based on SCA called ISSALD (Improved Leaders of Salp swarm Algorithm using Sine Cosine algorithm and Disrupt Operator) and ISSAF (Improved Followers of Salp swarm Algorithm using Sine Cosine algorithm). The updating process in consists to update the leaders (L) position by SCA and applying (Dop), whereas in ISSAF, the Dop is omitted and the position of followers is updated by SCA. Experimental results are evaluated on twenty datasets where four of them represent high dimensionality with a small number of instances. The obtained results show a good performance of ISSAFD in terms of accuracy, sensitivity, specificity, and the number of selected features in comparison with other metaheuristics (MH).
Spatiotemporal Modeling for Nonlinear Distributed Thermal Processes Based on KL Decomposition, MLP and LSTM Network

Estimation of absolute temperature distributions is crucial for many thermal processes in the nonlinear distributed parameter systems, such as predicting the curing temperature distribution of the chip, the temperature distribution of the catalytic rod, and so on. In this work, a spatiotemporal model based on the Karhunen-Loève (KL) decomposition, the multilayer perceptron (MLP), and the long short-term memory (LSTM) network, named KL-MLP-LSTM, is developed for estimating temperature distributions with a three-step procedure. Firstly, the infinite-dimensional model is transformed into a finite-dimensional model, where the KL decomposition method is used for dimension reduction and spatial basis functions extraction. Secondly, a novel MLP-LSTM hybrid time series model is constructed to deal with the two inherently coupled nonlinearities. Finally, the spatiotemporal temperature distribution model can be reconstructed through spatiotemporal synthesis. The effectiveness of the proposed model is validated by the data from a snap curing oven thermal process. Satisfactory agreement between the results of the current model and the other well-established model shows that the KL-MLP-LSTM model is reliable for estimating the temperature distributions during the thermal process.
The Architectural Implications of Facebook's DNN-Based Personalized Recommendation

The widespread application of deep learning has changed the landscape of computation in data centers. In particular, personalized recommendation for content ranking is now largely accomplished using deep neural networks. However, despite their importance and the amount of compute cycles they consume, relatively little research attention has been devoted to recommendation systems. To facilitate research and advance the understanding of these workloads, this paper presents a set of real-world, production-scale DNNs for personalized recommendation coupled with relevant performance metrics for evaluation. In addition to releasing a set of open-source workloads, we conduct in-depth analysis that underpins future system design and optimization for at-scale recommendation: Inference latency varies by 60% across three Intel server generations, batching and co-location of inference jobs can drastically improve latency-bounded throughput, and diversity across recommendation models leads to different optimization strategies.
Food supply chain in the era of Industry 4.0: blockchain technology implementation opportunities and impediments from the perspective of people, process, performance, and technology

The prevention of food loss throughout the supply chain, including manufacturers, has become a major challenge for a number of organisations. In addition, consumers are also increasingly interested in the authenticity of food and want to ensure that they receive the right quality of food. To address this issue, there is a need for reliable and robust tools to be available in the Industry 4.0 era that can trace the food throughout the supply chain from the farm through processing until it reaches the customer and, thus, ensure transparency. Using the people, process, and technology (PPT) model, this paper develops a blockchain-enabled food supply chain framework including the future opportunities and the present impediments based on the systematic literature review and semi-structured case interviews from the context of emerging economies. The study investigates the suitability of blockchain technology in resolving major challenges, such as traceability, trust, and accountability in the food industry. The study further paves the way for future researchers to address the technological and people-related challenges in the Industry 4.0 era to mitigate the emerging problems in the food sector. Interestingly, we did not find many issues in process- and performance-related aspects. The study offers empirical evidence of blockchain technology implementation in the Industry 4.0 era that opens up the discussion for future researchers and lists the potential threats.
Prediction of photovoltaic power output based on similar day analysis, genetic algorithm and extreme learning machine

Recently, many machine learning techniques have been successfully employed in photovoltaic (PV) power output prediction because of their strong non-linear regression capacities. However, single machine learning algorithm does not have stable prediction performance and sufficient generalization capability in the prediction of PV power output. In this work, a hybrid model (SDA-GA-ELM) based on extreme learning machine (ELM), genetic algorithm (GA) and customized similar day analysis (SDA) has been developed to predict hourly PV power output. In the SDA, Pearson correlation coefficient is employed to measure the similarity between different days based on five meteorological factors, and the data samples similar to those from the target forecast day are selected as the training set of ELM. This operation can effectively increase the number of useful samples and reduce the time consumption on training data. In the ELM, the optimal values of the hidden bias and the input weight are searched by GA to improve the prediction accuracy. The performance of the proposed forecast model is evaluated with coefficient of determination (R2), mean absolute error (MAE) and normalized root mean square error (nRMSE). The results show that the SDA-GA-ELM model has higher accuracy and stability in day-ahead PV power prediction.