Reviewerly/exHarmony huggingface dataset:
Deeply Exploit Visual and Language Information for Social Media Popularity Prediction

Social media popularity prediction task is to predict future attractiveness of new posts, which could be applied for online advertising, social recommendation, and demand prediction. Existing methods have explored multiple feature types to model the popularity prediction, including user profile, tag, space-time, category, and others. However, images and texts of social media posts, as important and primary information, are usually used by simple or insufficient processing. In this paper, we propose a method to deeply exploit visual and language information to explore the attractiveness of posts. Specifically, images are parsed from multiple perspectives including multi-modal semantic representation, perceptual image quality, and scene analysis. Different word-level and sentence-level semantic embedding are extracted from all available language texts including title, tags, concept and category. It makes social media popularity modeling more reliable with the powerful visual and language representation. Experimental results demonstrate the effectiveness of exploiting visual and language information by the proposed method, and we achieve new state-of-the-art results on the SMP Challenge at ACM Multimedia 2022.
Deep supervision and atrous inception-based U-Net combining CRF for automatic liver segmentation from CT

Due to low contrast and the blurred boundary between liver tissue and neighboring organs sharing similar intensity values, the problem of liver segmentation from CT images has not yet achieved satisfactory performance and remains a challenge. To alleviate these problems, we introduce deep supervision (DS) and atrous inception (AI) technologies with conditional random field (CRF) and propose three major improvements that are experimentally shown to have substantive and practical value. First, we replace the encoder's standard convolution with the residual block. Residual blocks can increase the depth of the network. Second, we provide an AI module to connect the encoder and decoder. AI allows us to obtain multi-scale features. Third, we incorporate the DS mechanism into the decoder. This helps to make full use of information of the shallow layers. In addition, we employ the Tversky loss function to balance the segmented and non-segmented regions and perform further refinement with a dense CRF. Finally, we extensively validate the proposed method on three public databases: LiTS17, 3DIRCADb, and SLiver07. Compared to the state-of-the-art methods, the proposed method achieved increased segmentation accuracy for the livers with low contrast and the fuzzy boundary between liver tissue and neighboring organs and is, therefore, more suited for automatic segmentation of these livers.
Unbiased Directed Object Attention Graph for Object Navigation

Object navigation tasks require agents to locate specific objects in unknown environments based on visual information. Previously, graph convolutions were used to implicitly explore the relationships between objects. However, due to differences in visibility among objects, it is easy to generate biases in object attention. Thus, in this paper, we propose a directed object attention (DOA) graph to guide the agent in explicitly learning the attention relationships between objects, thereby reducing the object attention bias. In particular, we use the DOA graph to perform unbiased adaptive object attention (UAOA) on the object features and unbiased adaptive image attention (UAIA) on the raw images, respectively. To distinguish features in different branches, a concise adaptive branch energy distribution (ABED) method is proposed. We assess our methods on the AI2-Thor dataset. Compared with the state-of-the-art (SOTA) method, our method reports 7.4%, 8.1% and 17.6% increase in success rate (SR), success weighted by path length (SPL) and success weighted by action efficiency (SAE), respectively.
Composite Photograph Harmonization with Complete Background Cues

Compositing portrait photographs or videos to novel backgrounds is an important application in computational photography. Seamless blending along boundaries and globally harmonic colors are two desired properties of the photo-realistic composition of foregrounds and new backgrounds. Existing works are dedicated to either foreground alpha matte generation or after-blending harmonization, leading to sub-optimal background replacement when putting foregrounds and backgrounds together. In this work, we unify the two objectives in a single framework to obtain realistic portrait image composites. Specifically, we investigate the usage of a target background and find that a complete background plays a vital role in both seamlessly blending and harmonization. We develop a network to learn the composition process given an imperfect alpha matte with appearance features extracted from the complete background to adjust color distribution. Our dedicated usage of a complete background enables realistic portrait image composition and also temporally stable results on videos. Extensive quantitative and qualitative experiments on both synthetic and real-world data demonstrate that our method achieves state-of-the-art performance.
A Transformer Based Approach for Activity Detection

Non-invasive physiological sensors allow for the collection of user-specific data in realistic environments. In this paper, using physiological data, we investigate the effectiveness of Convolutional Neural Network (CNN) based feature embeddings and Transformer architecture for the human activity recognition task. 1D-CNN representation is used for the heart rate, and 2D-CNN is used for short-term Fourier transformation of the accelerometer data. Post fusion, the feature is input into a transformer. The experiments are performed on the harAGE dataset. The findings indicate the discriminative ability of the feature-fusion on transformer-based architecture, and the method outperforms the harAGE baseline by an absolute 3.7%.
Deep Multi-Resolution Mutual Learning for Image Inpainting

Deep image inpainting methods have improved the inpainting performance greatly due to the powerful representation ability of deep learning. However, current deep inpainting networks still tend to produce unreasonable structures and blurry textures due to the ill-posed properties of the task, i.e., image inpainting is still a challenging topic. In this paper, we therefore propose a novel deep multi-resolution mutual learning (DMRML) strategy, which can fully explore the information from various resolutions. Specifically, we design a new image inpainting network, termed multi-resolution mutual network (MRM-Net), which takes the damaged images of different resolutions as input, then excavates and exploits the correlation among different resolutions to guide the image inpainting process. Technically, we designs two new modules called multi-resolution information interaction (MRII) and adaptive content enhancement (ACE). MRII aims at discovering the correlation of multiple resolutions and exchanging information, and ACE focuses on enhancing the contents using the interacted features. Note that we also present an memory preservation mechanism (MPM) to prevent from the information loss with the increasing layers. Extensive experiments on Paris Street View, Places2 and CelebA-HQ datasets demonstrate that our proposed MRM-Net can effectively recover the textures and structures, and performs favorably against other state-of-the-art methods.
No-reference Omnidirectional Image Quality Assessment Based on Joint Network

In panoramic multimedia applications, the perception quality of the omnidirectional content often comes from the observer's perception of the viewports and the overall impression after browsing. Starting from this hypothesis, this paper proposes a deep-learning based joint network to model the no-reference quality assessment of omnidirectional images. On the one hand, motivated by different scenarios that lead to different human understandings, a convolutional neural network (CNN) is devised to simultaneously encode the local quality features and the latent perception rules of different viewports, which are more likely to be noticed by the viewers. On the other hand, a recurrent neural network (RNN) is designed to capture the interdependence between viewports from their sequence representation, and then predict the impact of each viewport on the observer's overall perception. Experiments on two popular omnidirectional image quality databases demonstrate that the proposed method outperforms the state-of-the-art omnidirectional image quality metrics.
CrossNet: Boosting Crowd Counting with Localization

Generating high-quality density maps is a crucial step in crowd counting. It is obvious that exploiting the head location of the people can naturally highlight the crowded area and eliminate the interference of background noise. However, existing crowd counting methods are still tricky to reasonably use location in density generation. In this paper, a novel location-guided framework named CrossNet is proposed for crowd counting, which integrates location supervision into density maps through dual-branch joint training. First, a new branching network is proposed to localize the potential positions of pedestrians. With the help of supervision induced from the localization branch, Location Enhancement (LE) module is designed to obtain high-quality density maps by positioning foreground regions. Second, Adaptive Density Awareness Attention (ADAA) module is engaged to enhance localization accuracy, which can efficiently use the density of the counting branch to adaptively capture the error-prone dense areas of the location maps. Finally, Density Awareness Localization (DAL) loss is offered to allocate attention to the crowd density levels, which delivers more focus on regions with high densities and less concentration on areas with low densities. Extensive experiments conducted on four benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art approaches both in crowd counting and crowd localization.
Equivariant and Invariant Grounding for Video Question Answering

Video Question Answering (VideoQA) is the task of answering the natural language questions about a video. Producing an answer requires understanding the interplay across visual scenes in video and linguistic semantics in question. However, most leading VideoQA models work as black boxes, which make the visual-linguistic alignment behind the answering process obscure. Such black-box nature calls for visual explainability that reveals "What part of the video should the model look at to answer the question?". Only a few works present the visual explanations in a post-hoc fashion, which emulates the target model's answering process via an additional method.
A <sup>3</sup> GAN: Attribute-Aware Anonymization Networks for Face De-identification

Face de-identification (De-ID) removes face identity information in face images to avoid personal privacy leakage. Existing face De-ID breaks the raw identity by cutting out the face regions and recovering the corrupted regions via deep generators, which inevitably affect the generation quality and cannot control generation results according to subsequent intelligent tasks (eg., facial expression recognition). In this work, for the first attempt, we think the face De-ID from the perspective of attribute editing and propose an attribute-aware anonymization network (A3GAN) by formulating face De-ID as a joint task of semantic suppression and controllable attribute injection. Intuitively, the semantic suppression removes the identity-sensitive information in embeddings while the controllable attribute injection automatically edits the raw face along the attributes that benefit De-ID. To this end, we first design a multi-scale semantic suppression network with a novel suppressive convolution unit (SCU), which can remove the face identity along multi-level deep features progressively. Then, we propose an attribute-aware injective network (AINet) that can generate De-ID-sensitive attributes in a controllable way (i.e., specifying which attributes can be changed and which cannot) and inject them into the latent code of the raw face. Moreover, to enable effective training, we design a new anonymization loss to let the injected attributes shift far away from the original ones. We perform comprehensive experiments on four datasets covering four different intelligent tasks including face verification, face detection, facial expression recognition, and fatigue detection, all of which demonstrate the superiority of our face De-ID over state-of-the-art methods.
Energy-efficiency policies for decarbonising residential heating in Spain: A fuzzy cognitive mapping approach

Decarbonising residential heating poses a major challenge for the energy transition, but the policy attention devoted to and the support provided for it seem limited. There is a rather limited consideration of the perceptions of households, which play a large role in this sector. This in turn may affect the effectiveness of policy instruments. This study contributes to a better understanding of the various perceptions held by key stakeholders, in particular about what can be done to reduce residential heating bills and how. We use a participatory method based on Fuzzy Cognitive Mapping (FCM), which accounts for the behavioural complexity of the transition in residential heating and provides a better understanding of policy instruments in terms of stakeholder perception. We then test combinations of policy instruments based on expert opinion and identify the most effective ones based on their perceptions. Our analysis indicates that environmental education and information are perceived to reduce heating bills by a very significant amount. Taxes are perceived to be more effective than subsidies. We also show that policy packages are considered essential: when policy instruments are combined the effects perceived are cumulative, resulting in a much greater expected impact on heating bills and energy consumption.
kNN-MSDF: A Hardware Accelerator for k-Nearest Neighbors Using Most Significant Digit First Computation

Nearest Neighbors (k-NN) is a well-established algorithm for classification widely used in various machine learning applications. Although -NN has many advantages, it suffers severely from high latency and low throughput due to its computation-intensive nature. This paper suggests a novel, highly parallel approach based on Most Significant Digit First (MSDF) computing to accelerate k-NN algorithm where the Euclidean distance is used as the distance metric. The proposed method consists of three phases; in the first phase, subtract, and square functions are done on serially coming input data and provide the Most Significant Digit (MSD) of the distances. Then, the processed serially MSDF data are sorted, and finally, the label of test data is determined by majority voting. Having the value of MSD in advance provides the possibility of early termination of unnecessary computations. This approach leads to significantly higher performance and lower power consumption. Moreover, serial computation causes a lower area and memory footprint that paves the way to take advantage of the maximum number of parallel processing elements. The experimental results on an SoC platform indicate up to 88.3% improvement in terms of performance and energy consumption compared to the best previous designs.
RECO-HCON: A High-Throughput Reconfigurable Compact ASCON Processor for Trusted IoT

Statistics show that in 2030 the number of connected IoT devices will reach 25.44 billion, which can lead to the security breach in the back-end of high-performance computing clusters connected with the same network. Unfortunately, the current security primitives are not suitable algorithms to be implemented on physically constrained devices designed for IoT. Thus, the National Institute of Standard and Technology has announced a worldwide lightweight cryptographic competition (LWC) for securing tiny devices. This paper introduces a flexible, reconfigurable, and energy-efficient crypto-processor running one of the LWC finalist candidates - ASCON, which uses sponge construction that has fewer memory accesses that leads to less power consumption compared to other ones. The proposed processor is reconfigurable in a way both authenticated cipher (Encryption/decryption processes) and hash functions of ASCON are implemented in a six-mode compact fashion, covering a diversity of applications in the IoT spectrum. The design is developed in Chisel and evaluated in 28/32nm technology with commercial EDA tools. Evaluation results show that the proposed processor achieves the highest throughput while consuming 29% less power, operating at over 667 MHz. The design has also been implemented in Skywater 130nm technology node with the latest released OpenLane design flow to ensure an end-to-end open-source delivery of the IP.
Transductive Aesthetic Preference Propagation for Personalized Image Aesthetics Assessment

Personalized image aesthetics assessment (PIAA) aims at capturing individual aesthetic preference. Fine-tuning on personalized data has been proven to be effective in PIAA task. However, a fixed fine-tuning strategy may cause under/over-fitting on limited personal data and it also brings additional training cost. To alleviate these issues, we employ a meta learning-based Transductive Aesthetic Preference Propagation (TAPP-PIAA) algorithm under regression manner to substitute the fine-tuning strategy. Specifically, each user's data is regarded as a meta-task and spilt into support and query set. Then, we extract deep aesthetic features with a pre-trained generic image aesthetic assessment (GIAA) model. Next, we treat image features as graph nodes and their similarities as edge weights to construct an undirected nearest neighbor graph for inference. Instead of fine-tuning on support set, TAPP-PIAA propagates aesthetic preference from support to query set with a predefined propagation formula. Finally, to learn a generalizable aesthetic representation for various users, we optimize our TAPP-PIAA across different users with meta-learning framework. Experimental results indicate that our TAPP-PIAA can surpass the state-of-the-art methods on benchmark databases.
Keypoint-Guided Modality-Invariant Discriminative Learning for Visible-Infrared Person Re-identification

The visible-infrared person re-identification (VI-ReID) task aims to retrieve images of pedestrians across cameras with different modalities. In this task, the major challenges arise from two aspects: intra-class variations among images of the same identity, and cross-modality discrepancies between visible and infrared images. Existing methods mainly focus on the latter, attempting to alleviate the impact of modality discrepancy, which ignore the former issue of identity variations and achieve limited discrimination. To address both aspects, we propose a Keypoint-guided Modality-invariant Discriminative Learning (KMDL) method, which can simultaneously adapt to intra-ID variations and bridge the cross-modality gap. By introducing human keypoints, our method makes further exploration in the image space, feature space and loss constraints to solve the above issues. Specifically, considering the modality discrepancy in original images, we first design a Hue Jitter Augmentation (HJA) strategy, introducing the hue disturbance to alleviate color dependence in the input stage. To obtain discriminative fine-grained representation for retrieval, we design the Global-Keypoint Graph Module (GKGM) in feature space, which can directly extract keypoint-aligned features and mine relationships within global and keypoint embeddings. Based on these semantic local embeddings, we further propose the Keypoint-Aware Center (KAC) loss that can effectively adjust the feature distribution under the supervision of ID and keypoint to learn discriminative representation for the matching. Extensive experiments on SYSU-MM01 and RegDB datasets demonstrate the effectiveness of our KMDL method.
Collective Entity Alignment for Knowledge Fusion of Power Grid Dispatching Knowledge Graphs

Knowledge graphs (KGs) have been widely accepted as powerful tools for modeling the complex relationships between concepts and developing knowledge-based services. In recent years, researchers in the field of power systems have explored KGs to develop intelligent dispatching systems for increasingly large power grids. With multiple power grid dispatching knowledge graphs (PDKGs) constructed by different agencies, the knowledge fusion of different PDKGs is useful for providing more accurate decision supports. To achieve this, entity alignment that aims at connecting different KGs by identifying equivalent entities is a critical step. Existing entity alignment methods cannot integrate useful structural, attribute, and relational information while calculating entities' similarities and are prone to making many-to-one alignments, thus can hardly achieve the best performance. To address these issues, this paper proposes a collective entity alignment model that integrates three kinds of available information and makes collective counterpart assignments. This model proposes a novel knowledge graph attention network (KGAT) to learn the embeddings of entities and relations explicitly and calculates entities' similarities by adaptively incorporating the structural, attribute, and relational similarities. Then, we formulate the counterpart assignment task as an integer programming (IP) problem to obtain one-to-one alignments. We not only conduct experiments on a pair of PDKGs but also evaluate our model on three commonly used cross-lingual KGs. Experimental comparisons indicate that our model outperforms other methods and provides an effective tool for the knowledge fusion of PDKGs.
A construction of post quantum secure and signal leakage resistant authenticated key agreement protocol for mobile communication

Abstract Mobile communication is very important for an individual users, enterprisers, industries, businesses, and organizations. Since the nodes communicate through a public channel, security, and privacy become essential component for mobile communication. Both key exchange and authentication are two important security attributes to establish secure communication between two parties. In the past few years, various post‐quantum secure authentication protocols have been designed, but we have found that either some of the protocols are vulnerable to stolen smart card attack, invalid login, or signal leakage attack. In this article, we have discussed how the protocol Dharminder et al (Dharminder D, Chandran KP. J Ambient Intell Humaniz Comput. 2020; 11:4089‐4100.) is vulnerable to modified version of signal leakage attack, and sometimes called improved signal leakage attack. We have also suggested a remedy over the current existing protocol suffering from signal leakage attacks. The proposed protocol is secure against quantum attacks. We have proved the security of the proposed protocol under ring learning with errors assumption. The proposed authentication and key agreement needs only two messages exchange. We have also found the proposed is efficient in terms of computation with respect to relevant protocols.