Reviewerly/exHarmony huggingface dataset:
Joint intensity–gradient guided generative modeling for colorization

This paper proposes an iterative score-based generative model for solving the automatic colorization problem. Although unsupervised learning methods have shown the capability to generate plausible color, inadequate exploration of detailed information and data dimensions still limit the performance of the colorization model. Considering that the number of samples in score-based generative model has influence on estimating the target gradients and the gradient map possesses important latent information of the image, the inference process of the generative modeling is conducted in joint intensity–gradient domain for colorization. Specifically, a set of intensity–gradient formed high-dimensional tensors are trained, via the score matching, to attain the gradient of data distribution in joint intensity–gradient domain. As the score function is determined, data samples are generated by means of annealed Langevin dynamics, forming an iterative colorization procedure. Furthermore, the joint intensity–gradient constraint in data-fidelity term is proposed to limit the degree of freedom within generative model at the iterative colorization stage, thus being conducive to edge-preserving colorization effect. Experimental results conveyed the remarkable performance and diversity of our proposed method.
Aligning model outputs for class imbalanced non-IID federated learning

Federated Learning (FL) aims to generate a global shared model via collaborating with decentralized edge computing devices with privacy considerations. A significant challenge in FL is the non-IID data partitioning across heterogeneous devices, making the local update diverge a lot and difficult to aggregate. This diversity in local models is caused by the different posterior probability of samples when class distribution skews. Meanwhile, FL often faces imbalanced global data in practical scenarios. By analyzing the relationship between the samples' posterior probability in different data distributions, we propose a statistically principled probability-corrected loss to align the posterior probability when models are trained on heterogeneous clients. Additionally, we share fixed prototypes on each client to constrain the distribution of heterogeneous clients' features. Our approach can well handle non-IID FL with balanced and imbalanced global data. We combine our approach with existing FL algorithms and investigate it on common FL benchmarks. Abundant experimental results verify the superiorities of our methods.
Reconciling privacy and utility: an unscented Kalman filter-based framework for differentially private machine learning

Machine learning (ML) is extensively used in fields involving sensitive data, data holders are seeking to protect the privacy of data and build the ML models with high-quality utility. Differential privacy provides a feasible solution for them. However, there is a mutual constraint on the privacy and utility of the models under this solution. Therefore, how to improve (or even maximize) the utility of the model while preserving privacy becomes an urgent problem. To resolve this problem, we apply unscented Kalman filter (UKF) to various implementations of differential privacy (DP)-enabled ML (DPML). We propose a UKF-based DP-enabled ML (UKF-DPML) framework that achieves higher model utility with the given privacy budget $$\epsilon$$ . An evaluation module is included in the framework to ensure a fair estimation of DPML models. We validate the effectiveness of this framework through mathematical reasoning, followed by empirical evaluation of various implementations of UKF-DPML and DPML respectively. In the evaluation, we measure the ability of withstanding real-world privacy attacks and providing accurate classification, thus assessing the privacy and utility of the model. We conduct a range of privacy budgets and implementations on three datasets, each of which provides the same mathematical privacy guarantees. By measuring the resistance of UKF-DPML and DPML models to membership and attribute inference attacks and their classification accuracy, we obtain that applying UKF to the aggregates perturbed by DP noises results in higher utility with the same privacy budget and the effect of improved utility is related to the stage where UKF is applied.
An Energy-Efficient K-means Clustering FPGA Accelerator via Most-Significant Digit First Arithmetic

K-means clustering is the most well-known unsupervised learning method that partitions the input dataset into <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K$</tex> clusters based on the similarity between the data samples. In this paper, to achieve an energy-efficient implementation without sacrificing performance, we take advantage of massive parallelism and digit-level pipelining via FPGA and the most-significant digit first arithmetic. Having the result of the most-significant digits in advance provides the possibility of early termination for unnecessary computations and fetching just the required most-significant part of data points from memory. This early termination technique significantly increases performance and decreases energy consumption. Our experimental results from various datasets and comparisons with the state-of-the-art FPGA accelerators indicate that our proposed design has effectively reduced energy consumption without any performance loss.
Bandwidth Efficient Homomorphic Encrypted Matrix Vector Multiplication Accelerator on FPGA

Homomorphic Encryption (HE) is a promising solution to the increasing concerns of privacy in Machine Learning (ML) as it enables computations directly on encrypted data. However, it imposes significant overhead on the compute system and remains impractically slow. Prior works have proposed efficient FPGA implementations of basic HE primitives such as number theoretic transform (NTT), key switching, etc. Composing the primitives together to realize higher level ML computation is still a challenge due to the large data transfer overhead. In this work, we propose an efficient FPGA implementation of HE Matrix Vector Multiplication <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathbf{M}\times \mathbf{V})$</tex> , a key kernel in HE-based Machine Learning applications. By analyzing the data reuse characteristics and the encryption overhead of HE <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{M}\times \mathbf{V}$</tex> , we show that simply using the principles of unencrypted <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{M}\times \mathbf{V}$</tex> to design accelerators for HE <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{M}\times \mathbf{V}$</tex> can lead to a significant amount of DRAM data transfers. We tackle the computation and data transfer challenges by proposing a bandwidth efficient dataflow that is specially optimized for HE <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{M}\times \mathbf{V}$</tex> . We identify highly reused data entities in HE <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{M}\times \mathbf{V}$</tex> and efficiently utilize the on-chip SRAM to reduce the DRAM data transfers. To speed up the computation of HE <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{M}\times \mathbf{V}$</tex> , we exploit three types of parallelism: partial sum parallelism, residual polynomial parallelism and coefficient parallelism. Leveraging these innovations, we demonstrate the first FPGA accelerator for HE matrix vector multiplication. Evaluation on 7 HE <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{M}\times \mathbf{V}$</tex> benchmarks shows that our FPGA accelerator is up to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3.8\times$</tex> (GeoMean <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.8\times$</tex> ) faster compared to the 64-thread CPU implementation.
P3Net: PointNet-based Path Planning on FPGA

Path planning is of crucial importance for au-tonomous mobile robots, and comes with a wide range of real-world applications including transportation, surveillance, and rescue. Currently, its high computational complexity is a major bottleneck for the application on such resource-limited robots. As a promising and effective solution to tackle this issue, in this paper, we propose a novel learning-based method for 2D/3D path planning, P3Net (PointNet-based Path Planning Network), along with its resource-efficient implementation targeting Xilinx ZCU104 boards. Our proposal is built upon two improvements to the recently proposed MPNet: we use a parameter-efficient PointNet-based encoder network to extract high-fidelity obstacle features from a point cloud, in conjunction with a lightweight planning network to iteratively plan a path. Experimental results using 2D/3D datasets demonstrate that our FPGA-based P3Net performs significantly better than MPNet and even comparable to the state-of-the-art sampling-based methods such as BIT*. P3Net is able to plan near-optimal paths 6.24x-9.34x faster than MPNet, and eventually improves the success rate by up to 24.45%, while reducing the parameter size by 5.43x-32.32x. This enables the subsecond real-time performance in many cases and opens up a new research direction for the edge-based efficient path planning.
A Smart and Secured Approach for Children’s Health Monitoring Using Machine Learning Techniques Enhancing Data Privacy

Health monitoring of the infant and child is proposed to provide the assurance to the parents of the infant and child. Several existing features are initiated where the proximity is not identified. Everyday varied health conditions of the infant (0–4 years) and child (6–10 years) are attained to monitor their health constantly. Different vital parameters such as heart rate, blood pressure, temperature and emotions are detected using the sensors. Smart devices such as IR photodetector and wearable gadget sensors are proposed to monitor the health condition of the child and infant. To attain the movement of the infant and child, such as intrusion detection, the accelerometer is used and to monitor the kid, the camera is used to detect the intrusion. To monitor the child’s safety outside the environment, the GPS trackers with GSM are fixed to predict the necessary safety condition. IOT monitoring framework detects the zone and captures the information and sends notification to the parents. Different Machine learning algorithms are used to obtain the suitable output with accuracy and performance measures. The IOT monitoring and evaluated data are cross-validated to attain the precise results. The comparative result analysis of all the different algorithms is analysed. In this proposed architecture, the Blockchain technology is used to provide data security and to avoid non-repudiation services. Data reports are analysed where it is stored within the application to indicate whether the state is in a normal/abnormal condition, and then, the alert messages/calls are sent to the parents/guardian.
Enacting Special Education in a Digitalized School: Opening for New Understandings of a Digitalized Special Educational Practice

As society has gotten increasingly digitalized, schools have made extensive efforts to accommodate to that development. With the digitalization of school, the special education practice and the roles of special education needs coordinators (SENCOs) and special education teachers (SETs) change. However, there is a lack of research examining this transformation, a gap which this exploratory study opens up for new understandings of. This mixed method study explores the special education practice in a municipality in Sweden, one of the most digitalized countries in the world. The findings show that SENCOs and SETs have highly positive beliefs about digital tools in special education, but at the same time their frequency of use varies greatly. Also, the study identifies special educational activities engaging with digital technology, describing a work practice with complex expectations of the special education professionals also in the area of digital teaching and learning. Furthermore, the competencies considered as necessary for handling the expectations are discussed.
Clothes retrieval based on ResNet and cluster triplet loss

Clothes image retrieval is a task that retrieves exact or very similar clothes from a large clothes image gallery according to a given query image. The high deformability of clothes and similar designs make clothes retrieval very difficult. Convolutional neural networks perform well in the feature extraction field, but cannot extract sufficiently fine-grained features for clothes retrieval tasks. Therefore, a model based on ResNet was designed in this article, with several improvements made to it to ensure better feature discrimination. On the other hand, pair-based loss functions that are commonly used in image retrieval suffer from insufficient discrimination ability, difficulty in convergence, and time-consuming during training. Cluster triplet loss function is proposed in this article, which reduces the computational complexity in the training phase and enables the model to have a certain ability to resist noise labels. Experiments are completed on In-shop clothes retrieval benchmark and Consumer-to-shop clothes retrieval benchmark in Deepfashion, and stanford online products (SOP) to supplement. Our method can have Recall@1 improvements of 1.3–1.7% on all three datasets, surpassing the latest state-of-the-art methods.
Backdoor Attacks on Deep Neural Networks via Transfer Learning from Natural Images

Backdoor attacks are a serious security threat to open-source and outsourced development of computational systems based on deep neural networks (DNNs). In particular, the transferability of backdoors is remarkable; that is, they can remain effective after transfer learning is performed. Given that transfer learning from natural images is widely used in real-world applications, the question of whether backdoors can be transferred from neural models pretrained on natural images involves considerable security implications. However, this topic has not been evaluated rigorously in prior studies. Hence, in this study, we configured backdoors in 10 representative DNN models pretrained on a natural image dataset, and then fine-tuned the backdoored models via transfer learning for four real-world applications, including pneumonia classification from chest X-ray images, emergency response monitoring from aerial images, facial recognition, and age classification from images of faces. Our experimental results show that the backdoors generally remained effective after transfer learning from natural images, except for small DNN models. Moreover, the backdoors were difficult to detect using a common method. Our findings indicate that backdoor attacks can exhibit remarkable transferability in more realistic transfer learning processes, and highlight the need for the development of more advanced security countermeasures in developing systems using DNN models for sensitive or mission-critical applications.
Understanding User Preferences in Location-Based Social Networks via a Novel Self-Attention Mechanism

The check-in behaviors of users are ubiquitous in location-based social networks in urban living. Understanding user preferences is critical to improving the recommendation services of social platforms. In addition, great quality of recommendation is also beneficial to sustainable urban living since the user can easily find the point of interest (POI) to visit, which avoids unnecessary consumption, such as a longer time taken for searching or driving. To capture user preferences from their check-in behaviors, advanced methods transform historical records into graph structure data and further leverage graph deep learning-based techniques to learn user preferences. Despite their effectiveness, existing graph deep learning-based methods are limited to the capture of the deep graph’s structural information due to inherent limitations, such as the over-smoothing problem in graph neural networks, further leading to suboptimal performance. To address the above issues, we propose a novel method built on Transformer architecture named spatiotemporal aware transformer (STAT) via a novel graphically aware attention mechanism. In addition, a new temporally aware sampling strategy is developed to reduce the computational cost and enable STAT to deal with large graphs. Extensive experiments on real-world datasets have demonstrated the superiority of the STAT compared to state-of-the-art POI recommendation methods.
Review of Diessel (2019): The grammar network: How linguistic structure is shaped by language use

Abstract This article reviews The grammar network: How linguistic structure is shaped by language use 9781108671040
Enhancing the Confidentiality and Integrity of Uncertain Dynamic Data Workflows of B2C (Business-2-Consumers) Using Blockchain Technology

While B2C via cloud computing has grown rapidly, information security and trustworthy computation are still major issues in today’s industrial realm. As a result, the routine workflow process varies dynamically over the course of data transactions, resulting in uncertainty. Consumers’ private information becomes easily vulnerable due to such data uncertainty. Intruders/fake marketers may permeate deeply into their current profiles due to the Internet, which leads to the creation of fictitious online markets that, in turn, drive more demand. This evolving business interaction leverages new interlopes to target customers’ data more precisely. However, it is still lacking in terms of complete techniques for dealing with such issues. These days, uncertain B2C workflows are greatly aided by Blockchain technology, especially to most of its security-enhancing characteristics. Despite numerous existing studies that have recommended employing Blockchain to handle B2C records, it is necessary to employ the technology in such business sectors to offer secured record management, workflow and keep data with high confidentiality and integrity intact. In this study, we suggest a Distributed Decentralized Security Scheme (DDSS) that manages and secures uncertain datasets through Blockchain technology in the B2C sector. To improve the safety of data workflows, the proposed approach enables the user to verify the authenticity of the records they require. A smart contract or special chain-code is used to dictate the parameters of the suggested system, ensuring security. The findings and discussions explain that the suggested scheme is far more effective than the previous strategies and has a greater level of confidentiality and integrity strength.
Semi-Supervised Portrait Matting via the Collaboration of Teacher–Student Network and Adaptive Strategies

In the portrait matting domain, existing methods rely entirely on annotated images for learning. However, delicate manual annotations are time-consuming and there are few detailed datasets available. To reduce complete dependency on labeled datasets, we design a semi-supervised network (ASSN) with two kinds of innovative adaptive strategies for portrait matting. Three pivotal sub-modules are embedded in our architecture, including a static teacher network (S-TN), a static student network (S-SN), and an adaptive student network (A-SN). S-TN and S-SN are modules that need to be trained with a small number of high-quality labeled datasets. Moreover, A-SN and S-SN share the same module parameters. When processing unlabeled datasets, A-SN adopts the adaptive strategies designed by us to discard the dependence on labeled datasets. The adaptive strategies include: (i) An auxiliary adaption: The teacher network with complicated design not only provides alpha mattes for the adaptive student network but also transmits rough segmentation results and edge graphs as optimization reference standards. (ii) A self-adjusting adaption: The adaptive network can make self-supervised to the characteristics of different layers. In addition, we have produced a finely annotated dataset for scholars in the field. Compared with existing datasets, our dataset complements the following two types of data neglected in previous datasets: (i) Images taken by multiple people. (ii) Images under low light conditions.
The Effect of TOEIC Classes Applying Artificial Intelligence-based Adaptive Learning on Academic Achievement and Influencing Factors

Objectives The purpose of this study is to derive implications for the effective design and operation of TOEIC classes applying artificial intelligence (AI)-based adaptive learning by examining the effect of these classes on academic achievement and influencing factors. Methods To verify the effect, the differences in the pre and post TOEIC scores of all students who participated in the AI-based TOEIC classes at S University and groups divided by their initial TOEIC level were analyzed through paired sample t-tests. To examine the factors influencing the effect, the differences in TOEIC score improvement were analyzed in relation to student background variables through independent sample t-tests and one-way ANOVAs. AI learning factors that influence TOEIC score were examined through simple linear regression analyses. A total of 76 students’ data were used for these analyses. Results As a result of the analysis, it was discovered that the post-TOEIC scores of all participating students and the groups which had been divided based on the pre-TOEIC score (upper/lower) were significantly higher than those of the pre-TOEIC. No student background variables (gender, grade, department, or initial TOEIC level) affected the TOEIC score improvement. Also, it was found that, out of the AI learning factors (attendance rate, learning time, number of problems solved), the attendance rate and the number of problems solved had a significant effect on the TOEIC score. Conclusions TOEIC classes with AI-based adaptive learning that consider individual students’ learning levels and speed can contribute to the realization of customized education for students. To enhance the effectiveness of TOEIC classes using AI, this study suggests the application of blended learning to support students’ self-directed learning and the design and operation of classes that promote practical learning, such as periodic learning and solving many problems.