Reviewerly/exHarmony huggingface dataset:
NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation

Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusion by concatenating multi-modal MRIs at an early/middle stage of the network, which hardly explores non-linear dependencies between modalities. In this work, we propose a novel Nested Modality-Aware Transformer (NestedFormer) to explicitly explore the intra-modality and inter-modality relationships of multi-modal MRIs for brain tumor segmentation. Built on the transformer-based multi-encoder and single-decoder structure, we perform nested multi-modal fusion for high-level representations of different modalities and apply modality-sensitive gating (MSG) at lower scales for more effective skip connections. Specifically, the multi-modal fusion is conducted in our proposed Nested Modality-aware Feature Aggregation (NMaFA) module, which enhances long-term dependencies within individual modalities via a tri-orientated spatial-attention transformer, and further complements key contextual information among modalities via a cross-modality attention transformer. Extensive experiments on BraTS2020 benchmark and a private meningiomas segmentation (MeniSeg) dataset show that the NestedFormer clearly outperforms the state-of-the-arts. The code is available at https://github.com/920232796/NestedFormer .
A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation

BERT4Rec is an effective model for sequential recommendation based on the Transformer architecture. In the original publication, BERT4Rec claimed superiority over other available sequential recommendation approaches (e.g. SASRec), and it is now frequently being used as a state-of-the art baseline for sequential recommendation. However, not all subsequent publications confirmed its superiority and have proposed other models that were shown to outperform BERT4Rec in effectiveness. In this paper we systematically review all publications that compare BERT4Rec with another popular Transformer-based model, namely SASRec, and show that BERT4Rec results are not consistent within these publications. To understand the reasons behind this inconsistency, we analyse the available implementations of BERT4Rec and show that we fail to reproduce results of the original BERT4Rec publication when using their default configuration parameters. However, we are able to replicate the reported results with the original code if training for a much longer amount of time (up to 30x) compared to the default configuration. We also propose our own implementation of BERT4Rec based on the HuggingFace Transformers library, which we demonstrate replicates the originally reported results on 3 out 4 datasets, while requiring up to 95% less training time to converge. Overall, from our systematic review and detailed experiments, we conclude that BERT4Rec does indeed exhibit state-of-the-art effectiveness for sequential recommendation, but only when trained for a sufficient amount of time. Additionally, we show that our implementation can further benefit from adapting other Transformer architectures that are available in the HuggingFace Transformers library (e.g. using disentangled attention, as provided by DeBERTa, or larger hidden layer size cf. ALBERT). For example, on the MovieLens-1M dataset, we demonstrate that both these models can improve BERT4Rec performance by up to 9%. Moreover, we show that an ALBERT-based BERT4Rec model achieves better performance on that dataset than state-of-the-art results reported in the most recent publications.
Applying the structural causal model framework for observational causal inference in ecology

Abstract Ecologists are often interested in answering causal questions from observational data but generally lack the training to appropriately infer causation. When applying statistical analysis (e.g., generalized linear model) on observational data, common statistical adjustments can often lead to biased estimates between variables of interest due to processes such as confounding, overcontrol, and collider bias. To overcome these limitations, we present an overview of structural causal modeling (SCM), an emerging causal inference framework that can be used to determine cause‐and‐effect relationships from observational data. The SCM framework uses directed acyclic graphs (DAGs) to visualize researchers' assumptions about the causal structure of a system or process under study. Following this, a DAG‐based graphical rule known as the backdoor criterion can be applied to determine statistical adjustments (or lack thereof) required to determine causal relationships from observational data. In the presence of unobserved confounding variables, an additional rule called the frontdoor criterion can be employed to determine causal effects. Here, we use simulated ecological examples to review how the backdoor and frontdoor criteria can return accurate causal estimates between variables of interest, as well as how biases can arise when these criteria are not used. We further provide an overview of studies that have applied the SCM framework in ecology. SCM, along with its application of DAGs, has been widely used in other disciplines to make valid causal inferences from observational data. Their use in ecology holds tremendous potential for quantifying causal relationships and investigating a range of ecological questions without randomized experiments.
Zonotopic distributed fusion for nonlinear networked systems with bit rate constraint

In this paper, the distributed fusion estimation problem is studied for a class of nonlinear networked systems subject to unknown-but-bounded (UBB) noises. A bit rate constraint is introduced to quantify the limited bandwidth of the communication channel, under which a bit rate allocation protocol is further designed by solving a certain off-line optimization problem. Based on the received data from the network, several local extended-Kalman-type estimators are constructed and zonotopic sets confining local estimation errors are then obtained. By designing the local estimator parameters, the F-radii of the obtained zonotopic sets are minimized. Subsequently, with the calculated local estimates and zonotopic sets, a zonotopes-based distributed fusion estimator is put forward by means of the matrix-weighted fusion method, and the global zonotope (i.e., the zonotope encompassing the error between the system state and the fused estimate) is derived. Moreover, under the proposed zonotopes-based fusion framework, the distributed fusion estimators are designed based on, respectively, the scalar-weighted fusion method and the diagonal-matrix-weighted fusion method. Finally, the effectiveness of the proposed distributed fusion method is illustrated through a numerical example.
Multimodal attention-based deep learning for Alzheimer’s disease diagnosis

Alzheimer's disease (AD) is the most common neurodegenerative disorder with one of the most complex pathogeneses, making effective and clinically actionable decision support difficult. The objective of this study was to develop a novel multimodal deep learning framework to aid medical professionals in AD diagnosis.We present a Multimodal Alzheimer's Disease Diagnosis framework (MADDi) to accurately detect the presence of AD and mild cognitive impairment (MCI) from imaging, genetic, and clinical data. MADDi is novel in that we use cross-modal attention, which captures interactions between modalities-a method not previously explored in this domain. We perform multi-class classification, a challenging task considering the strong similarities between MCI and AD. We compare with previous state-of-the-art models, evaluate the importance of attention, and examine the contribution of each modality to the model's performance.MADDi classifies MCI, AD, and controls with 96.88% accuracy on a held-out test set. When examining the contribution of different attention schemes, we found that the combination of cross-modal attention with self-attention performed the best, and no attention layers in the model performed the worst, with a 7.9% difference in F1-scores.Our experiments underlined the importance of structured clinical data to help machine learning models contextualize and interpret the remaining modalities. Extensive ablation studies showed that any multimodal mixture of input features without access to structured clinical information suffered marked performance losses.This study demonstrates the merit of combining multiple input modalities via cross-modal attention to deliver highly accurate AD diagnostic decision support.
On the exploitation of the blockchain technology in the healthcare sector: A systematic review

The blockchain is a disruptive technology born in the last few years, which possible applications in different domains are being extensively studied. In this context, healthcare appears to be a very attractive application domain for the blockchain because, due to its characteristics, it can provide the necessary guarantees on the secure processing, sharing and management of sensitive patient data. In this paper, we perform a systematic review of the literature on the adoption of the blockchain technology in healthcare, focusing on applications implemented in real contexts. Our goal is to investigate the current state of the art in this specific field, emphasizing limitations and possible future developments. Publications extracted from Scopus, PubMed and Web of Science that satisfy some predetermined search criteria were collected by means of appropriate queries. These papers were analyzed and classified into five main categories, based on the specific sub-domain on which the applications were projected. The performed analysis highlighted that research activities are currently focused on data security and on the implementation of electronic health records through the Blockchain. On the other hand, some other areas are still under-explored, including that related to IoT or to the implementation of automated diagnosis systems.
Multiple medical image encryption algorithm based on scrambling of region of interest and diffusion of odd-even interleaved points

Due to the security requirement brought by the rapid development of electronic medical, this paper proposes an encryption algorithm for multiple medical images. The algorithm can not only encrypt grayscale medical images of any number and any size at the same time, but also has good encryption effect when applied to color images. Considering the characteristics of medical images, we design an encryption algorithm based on the region of interest (ROI). Firstly, extract the region of interest of the plaintext images and obtain the coordinates, calculate the hash value of the large image composed of all plaintext images. Set the coordinates and hash value as the secret key. This operation makes the whole encryption algorithm closely related to the plaintext images, which greatly enhances the ability to resist chosen plaintext attacks and improves security of the algorithm. In the process of encryption, chaotic sequences generated by Logistic-Tent chaotic system (LTS) are used to perform two scrambling and one diffusion, that is, pixel swapping based on the region of interest, Fisher-Yates scrambling and our newly proposed diffusion algorithm based on odd–even interleaved points. After testing and performance analysis, the algorithm can achieve good encryption effect, can resist various attacks, and has a higher security level and faster encryption speed.
Look Before You Leap: Improving Text-based Person Retrieval by Learning A Consistent Cross-modal Common Manifold

The core problem of text-based person retrieval is how to bridge the heterogeneous gap between multi-modal data. Many previous approaches contrive to learning a latent common manifold mapping paradigm following a \textbf{cross-modal distribution consensus prediction (CDCP)} manner. When mapping features from distribution of one certain modality into the common manifold, feature distribution of the opposite modality is completely invisible. That is to say, how to achieve a cross-modal distribution consensus so as to embed and align the multi-modal features in a constructed cross-modal common manifold all depends on the experience of the model itself, instead of the actual situation. With such methods, it is inevitable that the multi-modal data can not be well aligned in the common manifold, which finally leads to a sub-optimal retrieval performance. To overcome this \textbf{CDCP dilemma}, we propose a novel algorithm termed LBUL to learn a Consistent Cross-modal Common Manifold (C$^{3}$M) for text-based person retrieval. The core idea of our method, just as a Chinese saying goes, is to `\textit{san si er hou xing}', namely, to \textbf{Look Before yoU Leap (LBUL)}. The common manifold mapping mechanism of LBUL contains a looking step and a leaping step. Compared to CDCP-based methods, LBUL considers distribution characteristics of both the visual and textual modalities before embedding data from one certain modality into C$^{3}$M to achieve a more solid cross-modal distribution consensus, and hence achieve a superior retrieval accuracy. We evaluate our proposed method on two text-based person retrieval datasets CUHK-PEDES and RSTPReid. Experimental results demonstrate that the proposed LBUL outperforms previous methods and achieves the state-of-the-art performance.
Secure Electronic Voting (E-voting) System Based on Blockchain on Various Platforms

In today's world, online voting is becoming more popular. It has a lot of potential for lowering administrative expenses and increasing voter turnout. It eliminates the need for voters to travel to polling locations or print ballot papers because they can vote from anywhere with Internet criteria having been a struggle. In an E-voting system to provide security, we are using various cryptography techniques. In the existing E-voting system, all participants depend on a third-party entity for analysis and publishing of the voting results. To address this problem, many existing approaches use blockchain technology. These existing approaches face various problems like security issues and constraints on the number of voters. To address the two difficulties noted above, we offer a platform-agnostic, decentralized trust, and more secure E-voting mechanism that can be implemented on a platform-independent blockchain-based approach that supports smart contract execution.
NFTs and the Metaverse Revolution: Research Perspectives and Open Challenges

The Spatial Web vision aims towards the development of a multidimensional layer to the Web fabric where the physical layer is fused with some virtual environment. Advances in the development of computer-generated virtual words or a metaverse are pushing toward the realization of this vision. This chapter argues that distributed ledger technologies, like blockchain, can enable a ubiquitous and persistent architectural layer for the Spatial Web. The emergence of independent blockchain-based metaverses and decentralized gaming virtual world provides evidence on how the technology can enable this evolution. A blockchain layer offers a standardized infrastructure for tokenizing physical objects and creating their digital footprints in a manner that such tokens (fungible or non-fungible) can be digitally owned, exchanged, and shared. The chapter contributes an analysis of the current status of virtual worlds with a focus on the emerging blockchain-based metaverse and articulates the open challenges and research perspectives for the emerging space of blockchain-based metaverses.
Deep Learning Cascaded Feature Selection Framework for Breast Cancer Classification: Hybrid CNN with Univariate-Based Approach

With the help of machine learning, many of the problems that have plagued mammography in the past have been solved. Effective prediction models need many normal and tumor samples. For medical applications such as breast cancer diagnosis framework, it is difficult to gather labeled training data and construct effective learning frameworks. Transfer learning is an emerging strategy that has recently been used to tackle the scarcity of medical data by transferring pre-trained convolutional network knowledge into the medical domain. Despite the well reputation of the transfer learning based on the pre-trained Convolutional Neural Networks (CNN) for medical imaging, several hurdles still exist to achieve a prominent breast cancer classification performance. In this paper, we attempt to solve the Feature Dimensionality Curse (FDC) problem of the deep features that are derived from the transfer learning pre-trained CNNs. Such a problem is raised due to the high space dimensionality of the extracted deep features with respect to the small size of the available medical data samples. Therefore, a novel deep learning cascaded feature selection framework is proposed based on the pre-trained deep convolutional networks as well as the univariate-based paradigm. Deep learning models of AlexNet, VGG, and GoogleNet are randomly selected and used to extract the shallow and deep features from the INbreast mammograms, whereas the univariate strategy helps to overcome the dimensionality curse and multicollinearity issues for the extracted features. The optimized key features via the univariate approach are statistically significant (p-value ≤ 0.05) and have good capability to efficiently train the classification models. Using such optimal features, the proposed framework could achieve a promising evaluation performance in terms of 98.50% accuracy, 98.06% sensitivity, 98.99% specificity, and 98.98% precision. Such performance seems to be beneficial to develop a practical and reliable computer-aided diagnosis (CAD) framework for breast cancer classification.
A ResNet deep learning based facial recognition design for future multimedia applications

• A real time facial emotions recognition application using deep learning technology. • Facial emotions detection even facial images having face mask and spectacles. • Facial expression features extraction during training process with ResNet technology. • Deep learning based continues miniaturization of facial emotions and tracking. • Classification and feature extraction mechanism can help the Facial emotions. CNN deep-learning models are the most trusted computational approaches in industrial, medical, and intelligent applications. In this research work, a deep micro-facial emotion has been recognised to employ CNN approach. Multi-class feature extraction has been initiated by the 2D-ResNet convolutional neural network. In order to detect the maskable images of facial emotions, a 2D-ResNet CNN multi-class classifier has been proposed. This novel model has reduced overfitting and controversial problems, which are verified on the Python 3.7.0 software tool. The public dataset JAFFE (open source) has been used to train the 2D-ResNet deep learning model. The flask, mysql, fastai, and jsonify python packages are compatible and feasible with the 2D-ResNet Deep Learning architecture. The proposed method practices better performance metrics in terms of accuracy of 99.3%, recall of 99.12%, F1 score of 0.98% and sensitivity of 99.16%, which are figured out from the confusion matrix, where only 5 failures have been observed when tested with 1 lakh images.
SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization

Fine-grained visual categorization (FGVC) aims at recognizing objects from similar subordinate categories, which is challenging and practical for human's accurate automatic recognition needs. Most FGVC approaches focus on the attention mechanism research for discriminative regions mining while neglecting their interdependencies and composed holistic object structure, which are essential for model's discriminative information localization and understanding ability. To address the above limitations, we propose the Structure Information Modeling Transformer (SIM-Trans) to incorporate object structure information into transformer for enhancing discriminative representation learning to contain both the appearance information and structure information. Specifically, we encode the image into a sequence of patch tokens and build a strong vision transformer framework with two well-designed modules: (i) the structure information learning (SIL) module is proposed to mine the spatial context relation of significant patches within the object extent with the help of the transformer's self-attention weights, which is further injected into the model for importing structure information; (ii) the multi-level feature boosting (MFB) module is introduced to exploit the complementary of multi-level features and contrastive learning among classes to enhance feature robustness for accurate fine-grained visual categorization. The proposed two modules are light-weighted and can be plugged into any transformer network and trained end-to-end easily, which only depends on the attention weights that come with the vision transformer itself. Extensive experiments and analyses demonstrate that the proposed SIM-Trans achieves state-of-the-art performance on fine-grained visual categorization benchmarks. The code will be available at https://github.com/PKU-ICST-MIPL/SIM-Trans_ACMMM2022.
Magic ELF: Image Deraining Meets Association Learning and Transformer

Convolutional neural network (CNN) and Transformer have achieved great success in multimedia applications. However, little effort has been made to effectively and efficiently harmonize these two architectures to satisfy image deraining. This paper aims to unify these two architectures to take advantage of their learning merits for image deraining. In particular, the local connectivity and translation equivariance of CNN and the global aggregation ability of self-attention (SA) in Transformer are fully exploited for specific local context and global structure representations. Based on the observation that rain distribution reveals the degradation location and degree, we introduce degradation prior to help background recovery and accordingly present the association refinement deraining scheme. A novel multi-input attention module (MAM) is proposed to associate rain perturbation removal and background recovery. Moreover, we equip our model with effective depth-wise separable convolutions to learn the specific feature representations and trade off computational complexity. Extensive experiments show that our proposed method (dubbed as ELF) outperforms the state-of-the-art approach (MPRNet) by 0.25 dB on average, but only accounts for 11.7% and 42.1% of its computational cost and parameters.
Feature and Semantic Views Consensus Hashing for Image Set Classification

Image set classification (ISC) has always been an active topic, primarily due to the fact that image set can provide more comprehensive information to describe a subject. However, the existing ISC methods face two problems: (1) The high computational cost prohibits these methods from being applied into median or large-scale applications; (2) the consensus information between feature and semantic representation of image set are largely ignored. To overcome these issues, in this paper, we propose a novel ISC method, termly feature and semantic views consensus hashing (FSVCH). Specifically, a kernelized bipartite graph is constructed to capture the nonlinear structure of data, and then two-views (\ie feature and semantic) consensus hashing learning (TCHL) is proposed to obtain a shared hidden consensus information. Meanwhile, for robust out-of-sample prediction purpose, we further propose TCHL guided optimal hash function inversion (TGHI) to learn a high-quality general hash function. Afterwards, hashing rotating (HR) is employed to obtain a more approximate real-valued hash solution. A large number of experiments show that FSVCH remarkably outperforms comparison methods on three benchmark datasets, in term of running time and classification performance. Experimental results also indicate that FSVCH can be scalable to median or large-scale ISC task.