Reviewerly/exHarmony huggingface dataset:
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs

Action recognition has typically treated actions and activities as monolithic events that occur in videos. However, there is evidence from Cognitive Science and Neuroscience that people actively encode activities into consistent hierarchical part structures. However, in Computer Vision, few explorations on representations that encode event partonomies have been made. Inspired by evidence that the prototypical unit of an event is an action-object interaction, we introduce Action Genome, a representation that decomposes actions into spatio-temporal scene graphs. Action Genome captures changes between objects and their pairwise relationships while an action occurs. It contains 10K videos with 0.4M objects and 1.7M visual relationships annotated. With Action Genome, we extend an existing action recognition model by incorporating scene graphs as spatio-temporal feature banks to achieve better performance on the Charades dataset. Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42.7% mAP using as few as 10 examples. Finally, we benchmark existing scene graph models on the new task of spatio-temporal scene graph prediction.
Constructing dummy query sequences to protect location privacy and query privacy in location-based services

Location-based services (LBS) have become an important part of people’s daily life. However, while providing great convenience for mobile users, LBS result in a serious problem on personal privacy, i.e., location privacy and query privacy. However, existing privacy methods for LBS generally take into consideration only location privacy or query privacy, without considering the problem of protecting both of them simultaneously. In this paper, we propose to construct a group of dummy query sequences, to cover up the query locations and query attributes of mobile users and thus protect users’ privacy in LBS. First, we present a client-based framework for user privacy protection in LBS, which requires not only no change to the existing LBS algorithm on the server-side, but also no compromise to the accuracy of a LBS query. Second, based on the framework, we introduce a privacy model to formulate the constraints that ideal dummy query sequences should satisfy: (1) the similarity of feature distribution, which measures the effectiveness of the dummy query sequences to hide a true user query sequence; and (2) the exposure degree of user privacy, which measures the effectiveness of the dummy query sequences to cover up the location privacy and query privacy of a mobile user. Finally, we present an implementation algorithm to well meet the privacy model. Besides, both theoretical analysis and experimental evaluation demonstrate the effectiveness of our proposed approach, which show that the location privacy and attribute privacy behind LBS queries can be effectively protected by the dummy queries generated by our approach.
Handling Information Loss of Graph Neural Networks for Session-based Recommendation

Recently, graph neural networks (GNNs) have gained increasing popularity due to their convincing performance in various applications. Many previous studies also attempted to apply GNNs to session-based recommendation and obtained promising results. However, we spot that there are two information loss problems in these GNN-based methods for session-based recommendation, namely the lossy session encoding problem and the ineffective long-range dependency capturing problem. The first problem is the lossy session encoding problem. Some sequential information about item transitions is ignored because of the lossy encoding from sessions to graphs and the permutation-invariant aggregation during message passing. The second problem is the ineffective long-range dependency capturing problem. Some long-range dependencies within sessions cannot be captured due to the limited number of layers. To solve the first problem, we propose a lossless encoding scheme and an edge-order preserving aggregation layer based on GRU that is dedicatedly designed to process the losslessly encoded graphs. To solve the second problem, we propose a shortcut graph attention layer that effectively captures long-range dependencies by propagating information along shortcut connections. By combining the two kinds of layers, we are able to build a model that does not have the information loss problems and outperforms the state-of-the-art models on three public datasets.
Convergence of Blockchain and Edge Computing for Secure and Scalable IIoT Critical Infrastructures in Industry 4.0

Critical infrastructure systems are vital to underpin the functioning of a society and economy. Due to the ever-increasing number of Internet-connected Internet-of-Things (IoT)/Industrial IoT (IIoT), and the high volume of data generated and collected, security and scalability are becoming burning concerns for critical infrastructures in industry 4.0. The blockchain technology is essentially a distributed and secure ledger that records all the transactions into a hierarchically expanding chain of blocks. Edge computing brings the cloud capabilities closer to the computation tasks. The convergence of blockchain and edge computing paradigms can overcome the existing security and scalability issues. In this article, we first introduce the IoT/IIoT critical infrastructure in industry 4.0, and then we briefly present the blockchain and edge computing paradigms. After that, we show how the convergence of these two paradigms can enable secure and scalable critical infrastructures. Then, we provide a survey on the state of the art for security and privacy and scalability of IoT/IIoT critical infrastructures. A list of potential research challenges and open issues in this area is also provided, which can be used as useful resources to guide future research.
Knowledge base graph embedding module design for Visual question answering model

In this paper, a knowledge base graph embedding module is constructed to extend the versatility of knowledge-based VQA (Visual Question Answering) models. The knowledge base graph embedding module constructed in this paper extracts core entities from images and text, and maps them as knowledge base entities, then extracts the sub-graphs closely related to the core entities, and converts the sub-graphs into low-dimensional vectors to realize sub-graph embedding. In order to achieve good subgraph embedding, we first extracted two experimental knowledge bases with rich semantics from DBpedia: DBV and DBA. Based on these two knowledge bases, this paper selects several excellent models in knowledge base embedding as test models, including SE (structured embedding),SME(semantic matching energy function), and TransE model to produce link prediction. The results show that there is a clear correspondence between the entities of the DBV, which can achieve excellent node embedding. And the TransE model can achieve a good knowledge base embedding, so we built the knowledge base graph embedding module based on TransE. And then we construct a VQA model (KBSN) based on the knowledge base graph embedding. Experimental results on VQA2.0 and KB-VQA data sets prove that the knowledge base graph embedding module improves the accuracy.
Deep Semantic Feature Learning for Software Defect Prediction

Software defect prediction, which predicts defective code regions, can assist developers in finding bugs and prioritizing their testing efforts. Traditional defect prediction features often fail to capture the semantic differences between different programs. This degrades the performance of the prediction models built on these traditional features. Thus, the capability to capture the semantics in programs is required to build accurate prediction models. To bridge the gap between semantics and defect prediction features, we propose leveraging a powerful representation-learning algorithm, deep learning, to learn the semantic representations of programs automatically from source code files and code changes. Specifically, we leverage a deep belief network (DBN) to automatically learn semantic features using token vectors extracted from the programs' abstract syntax trees (AST) (for file-level defect prediction models) and source code changes (for change-level defect prediction models). We examine the effectiveness of our approach on two file-level defect prediction tasks (i.e., file-level within-project defect prediction and file-level cross-project defect prediction) and two change-level defect prediction tasks (i.e., change-level within-project defect prediction and change-level cross-project defect prediction). Our experimental results indicate that the DBN-based semantic features can significantly improve the examined defect prediction tasks. Specifically, the improvements of semantic features against existing traditional features (in F1) range from 2.1 to 41.9 percentage points for file-level within-project defect prediction, from 1.5 to 13.4 percentage points for file-level cross-project defect prediction, from 1.0 to 8.6 percentage points for change-level within-project defect prediction, and from 0.6 to 9.9 percentage points for change-level cross-project defect prediction.
Cloud-assisted secure eHealth systems for tamper-proofing EHR via blockchain

The wide deployment of cloud-assisted electronic health (eHealth) systems has already shown great benefits in managing electronic health records (EHRs) for both medical institutions and patients. However, it also causes critical security concerns. Since once a medical institution generates and outsources the patients’ EHRs to cloud servers, patients would not physically own their EHRs but the medical institution can access the EHRs as needed for diagnosing, it makes the EHRs integrity protection a formidable task, especially in the case that a medical malpractice occurs, where the medical institution may collude with the cloud server to tamper with the outsourced EHRs to hide the medical malpractice. Traditional cryptographic primitives for the purpose of data integrity protection cannot be directly adopted because they cannot ensure the security in the case of collusion between the cloud server and medical institution. In this paper, a secure cloud-assisted eHealth system is proposed to protect outsourced EHRs from illegal modification by using the blockchain technology (blockchain-based currencies, e.g., Ethereum). The key idea is that the EHRs only can be outsourced by authenticated participants and each operation on outsourcing EHRs is integrated into the public blockchain as a transaction. Since the blockchain-based currencies provide a tamper-proofing way to conduct transactions without a central authority, the EHRs cannot be modified after the corresponding transaction is recorded into the blockchain. Therefore, given outsourced EHRs, any participant can check their integrity by checking the corresponding transaction. Security analysis and performance evaluation demonstrate that the proposed system can provide a strong security guarantee with a high efficiency.
Development of an interactive web-based tool to conduct and interrogate meta-analysis of diagnostic test accuracy studies: MetaDTA

Recommended statistical methods for meta-analysis of diagnostic test accuracy studies require relatively complex bivariate statistical models which can be a barrier for non-statisticians. A further barrier exists in the software options available for fitting such models. Software accessible to non-statisticians, such as RevMan, does not support the fitting of bivariate models thus users must seek statistical support to use R, Stata or SAS. Recent advances in web technologies make analysis tool creation much simpler than previously. As well as accessibility, online tools can allow tailored interactivity not found in other packages allowing multiple perspectives of data to be displayed and information to be tailored to the user's preference from a simple interface. We set out to: (i) Develop a freely available web-based "point and click" interactive tool which allows users to input their DTA study data and conduct meta-analyses for DTA reviews, including sensitivity analyses. (ii) Illustrate the features and benefits of the interactive application using an existing DTA meta-analysis for detecting dementia.To create our online freely available interactive application we used the existing R packages lme4 and Shiny to analyse the data and create an interactive user interface respectively.MetaDTA, an interactive online application was created for conducting meta-analysis of DTA studies. The user interface was designed to be easy to navigate having different tabs for different functions. Features include the ability for users to enter their own data, customise plots, incorporate quality assessment results and quickly conduct sensitivity analyses. All plots produced can be exported as either .png or .pdf files to be included in report documents. All tables can be exported as .csv files.MetaDTA, is a freely available interactive online application which meta-analyses DTA studies, plots the summary ROC curve, incorporates quality assessment results and allows for sensitivity analyses to be conducted in a timely manner. Due to the rich feature-set and user-friendliness of the software it should appeal to a wide audience including those without specialist statistical knowledge. We encourage others to create similar applications for specialist analysis methods to encourage broader uptake which in-turn could improve research quality.
Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor

Convolutional neural networks (CNNs) have shown remarkable performance in various real-world applications. Unfortunately, the promising performance of CNNs can be achieved only when their architectures are optimally constructed. The architectures of state-of-the-art CNNs are typically handcrafted with extensive expertise in both CNNs and the investigated data, which consequently hampers the widespread adoption of CNNs for less experienced users. Evolutionary deep learning (EDL) is able to automatically design the best CNN architectures without much expertise. However, the existing EDL algorithms generally evaluate the fitness of a new architecture by training from scratch, resulting in the prohibitive computational cost even operated on high-performance computers. In this paper, an end-to-end offline performance predictor based on the random forest is proposed to accelerate the fitness evaluation in EDL. The proposed performance predictor shows the promising performance in term of the classification accuracy and the consumed computational resources when compared with 18 state-of-the-art peer competitors by integrating into an existing EDL algorithm as a case study. The proposed performance predictor is also compared with the other two representatives of existing performance predictors. The experimental results show the proposed performance predictor not only significantly speeds up the fitness evaluations but also achieves the best prediction among the peer performance predictors.
Deep-Person: Learning discriminative deep features for person Re-Identification

Person re-identification (Re-ID) requires discriminative features focusing on the full person to cope with inaccurate person bounding box detection, background clutter, and occlusion. Many recent person Re-ID methods attempt to learn such features describing full person details via part-based feature representation. However, the spatial context between these parts is ignored for the independent extractor on each separate part. In this paper, we propose to apply Long Short-Term Memory (LSTM) in an end-to-end way to model the pedestrian, seen as a sequence of body parts from head to foot. Integrating the contextual information strengthens the discriminative ability of local feature aligning better to full person. We also leverage the complementary information between local and global feature. Furthermore, we integrate both identification task and ranking task in one network, where a discriminative embedding and a similarity measurement are learned concurrently. This results in a novel three-branch framework named Deep-Person, which learns highly discriminative features for person Re-ID. Experimental results demonstrate that Deep-Person outperforms the state-of-the-art methods by a large margin on three challenging datasets including Market-1501, CUHK03, and DukeMTMC-reID.
Variable Rate Deep Image Compression With a Conditional Autoencoder

In this paper, we propose a novel variable-rate learned image compression framework with a conditional autoencoder. Previous learning-based image compression methods mostly require training separate networks for different compression rates so they can yield compressed images of varying quality. In contrast, we train and deploy only one variable-rate image compression network implemented with a conditional autoencoder. We provide two rate control parameters, i.e., the Lagrange multiplier and the quantization bin size, which are given as conditioning variables to the network. Coarse rate adaptation to a target is performed by changing the Lagrange multiplier, while the rate can be further fine-tuned by adjusting the bin size used in quantizing the encoded representation. Our experimental results show that the proposed scheme provides a better rate-distortion trade-off than the traditional variable-rate image compression codecs such as JPEG2000 and BPG. Our model also shows comparable and sometimes better performance than the state-of-the-art learned image compression models that deploy multiple networks trained for varying rates.
White blood cells detection and classification based on regional convolutional neural networks

White blood cells (WBC) are important parts of our immune system and they protect our body against infections by eliminating viruses, bacteria, parasites and fungi. There are five types of WBC. These are called Lymphocytes, Monocytes, Eosinophils, Basophils and Neutrophils. The number of WBC types and the total number of WBCs provide important information about our health status. Diseases such as leukemia, AIDS, autoimmune diseases, immune deficiencies, blood diseases can be diagnosed based on the number of WBCs. In this study, a computer-aided automated system that can easily identify and locate WBC types in blood images has been proposed. Current blood test devices usually detect WBCs with traditional image processing methods such as preprocessing, segmentation, feature extraction, feature selection and classification. Deep learning methodology is superior to traditional image processing methods in literature. In addition, traditional methods require the appearance of the whole object to be able to recognize objects. Contrary to traditional methods, convolutional neural networks (CNN), a deep learning architecture, can extract features from a part of an object and perform object recognition. In this case, a CNN-based system shows a higher performance in recognizing partially visible cells for reasons such as overlap or only partial visibility of the image. Therefore, it has been the motivation of this study to increase the performance of existing blood test devices with deep learning method. Blood cells have been identified and classified by Regional Based Convolutional Neural Networks. Designed architectures have been trained and tested by combining BCCD data set and LISC data set. Regional Convolutional Neural Networks (R – CNN) has been used as a methodology. In this way, different cell types within the same image have been classified simultaneously with a detector. While training CNN which is the basis of R – CNN architecture; AlexNet, VGG16, GoogLeNet, ResNet50 architectures have been tested with full learning and transfer learning. At the end of the study, the system has showed 100% success in determining WBC cells. ResNet50, one of the CNN architectures, has showed the best performance with transfer learning. Cell types of Lymphocyte were determined with 99.52% accuracy rate, Monocyte with 98.40% accuracy rate, Basophil with 98.48% accuracy rate, Eosinophil with 96.16% accuracy rate and Neutrophil with 95.04% accuracy rate.
A novel image encryption scheme based on conservative hyperchaotic system and closed-loop diffusion between blocks

Dissipative chaotic systems have been widely used in digital image encryption schemes in the past 20 years. However, compared to conservative chaotic systems, the dissipative chaotic systems have attractors thus attacker can reconstruct the chaotic systems by reconstructing the attractors. Therefore, the conservative chaotic systems are more suitable in chaos-based encryption system because they have no attractors thus can avoid the reconstructing attacks. Based on this, an image encryption scheme based on conservative hyper-chaotic system and closed-loop diffusion between blocks is proposed in this paper. On the one hand, the conservative hyperchaotic system has strong pseudo-randomness and anti-reconstruction attack property. On the other hand, different from the existing closed-loop diffusion schemes which change pixel values one by one until the last pixel value is changed, the proposed closed-loop diffusion method cannot only generate the first ciphertext and other ciphertext blocks but also update the first ciphertext block using the other generated ciphertext block information. In addition, the key streams are related to plaintext and ciphertext. Consequently, the key, plaintext and ciphertext form an organic whole to ensure the sensitivity of the encryption system. Moreover, simulation results and analysis show that the encryption scheme has strong security and excellent performance.
Plundervolt: Software-based Fault Injection Attacks against Intel SGX

Dynamic frequency and voltage scaling features have been introduced to manage ever-growing heat and power consumption in modern processors. Design restrictions ensure frequency and voltage are adjusted as a pair, based on the current load, because for each frequency there is only a certain voltage range where the processor can operate correctly. For this purpose, many processors (including the widespread Intel Core series) expose privileged software interfaces to dynamically regulate processor frequency and operating voltage.In this paper, we demonstrate that these privileged interfaces can be reliably exploited to undermine the system's security. We present the Plundervolt attack, in which a privileged software adversary abuses an undocumented Intel Core voltage scaling interface to corrupt the integrity of Intel SGX enclave computations. Plundervolt carefully controls the processor's supply voltage during an enclave computation, inducing predictable faults within the processor package. Consequently, even Intel SGX's memory encryption/authentication technology cannot protect against Plundervolt. In multiple case studies, we show how the induced faults in enclave computations can be leveraged in real-world attacks to recover keys from cryptographic algorithms (including the AES-NI instruction set extension) or to induce memory safety vulnerabilities into bug-free enclave code. We finally discuss why mitigating Plundervolt is not trivial, requiring trusted computing base recovery through microcode updates or hardware changes.