Reviewerly/exHarmony huggingface dataset:
On Conditioning GANs to Hierarchical Ontologies

The recent success of Generative Adversarial Networks (GAN) is a result of their ability to generate high quality images from a latent vector space. An important application is the generation of images from a text description, where the text description is encoded and further used in the conditioning of the generated image. Thus the generative network has to additionally learn a mapping from the text latent vector space to a highly complex and multi-modal image data distribution, which makes the training of such models challenging. To handle the complexities of fashion image and meta data, we propose Ontology Generative Adversarial Networks (O-GANs) for fashion image synthesis that is conditioned on an hierarchical fashion ontology in order to improve the image generation fidelity. We show that the incorporation of the ontology leads to better image quality as measured by Fr\'{e}chet Inception Distance and Inception Score. Additionally, we show that the O-GAN achieves better conditioning results evaluated by implicit similarity between the text and the generated image.
Roadmap on Material-Function Mapping for Photonic-Electronic Hybrid Neural Networks

Driven by machine-learning tasks neural networks have demonstrated useful capabilities as nonlinear hypothesis classifiers. The underlying technologies performing the dot product multiplication, the summation, and the nonlinear thresholding on the input data in electronics, however, are limited by the same capacitive challenges known from electronic integrated circuits. The optical domain, in contrast, provides low delay interconnectivity suitable for such node distributed non Von Neumann architectures relying on dense node to node communication. Thus, once the neural network's weights are set, the delay of the network is just given by the time of flight of the photon, which is in the picosecond range for photonic integrated circuits. However, the functionality of memory for storing the trained weights does not exists in optics, thus demanding a fresh look to explore synergies between photonics and electronics in neural networks. Here we provide a roadmap to pave the way for emerging hybridized photonic electronic neural networks by taking a detailed look into a single node's perceptron, discussing how it can be realized in hybrid photonic electronic heterogeneous technologies. We show that a set of materials exist that exploit synergies with respect to a number of constrains including electronic contacts, memory functionality, electrooptic modulation, optical nonlinearity, and device packaging. We find that the material ITO, in particular, could provide a viable path for both the perceptron weights and the nonlinear activation function, while simultaneously being a foundry process near material. We finally identify a number of challenges that, if solved, could accelerate the adoption of such heterogeneous integration strategies of emerging memory materials into integrated photonics platforms for real time responsive neural networks.
Social Relation Recognition in Egocentric Photostreams

This paper proposes an approach to automatically categorize the social interactions of a user wearing a photo-camera 2fpm, by relying solely on what the camera is seeing. The problem is challenging due to the overwhelming complexity of social life and the extreme intra-class variability of social interactions captured under unconstrained conditions. We adopt the formalization proposed in Bugental's social theory, that groups human relations into five social domains with related categories. Our method is a new deep learning architecture that exploits the hierarchical structure of the label space and relies on a set of social attributes estimated at frame level to provide a semantic representation of social interactions. Experimental results on the new EgoSocialRelation dataset demonstrate the effectiveness of our proposal.
The Use of Formal Approach and Techniques Applied to Understand and Improve the Blockchain

Abstract: With data volumes expected to expand exponentially in the next years, protecting that data is critical. To ensure the reliability of the system, we use cutting-edge tools like blockchain. It wasn't until the advent of bitcoin, however, that a sizable portion of the population took notice of blockchain technology. Since its inception, blockchain technology has been used in the academic community and by businesses. What this means is that the information is being stored in a way that makes tampering and unauthorized access very difficult. Blocks, an abbreviation for "blockchain," a digital public ledger, are used to record monetary transactions. All nodes will eventually agree on the same sequence of block attachments, and this is achieved by a process termed "consensus." Learning about the inner workings of each blockchain and the rationale for its unique style of operation is possible via a thorough examination of these algorithms. In this study, I examined and contrasted the different consensus mechanisms now in use with consensus protocols, analyzing their relative relevance in blockchain development. The consensus algorithm behind any specific blockchain-based system is crucial to the reliability and safety of that system. Formal approaches may be used to both increase users' trust in blockchain-based systems and design thoroughly tested and trustworthy solutions. To achieve this level of confidence, consensus methods must be effective. Formal modeling is developing a system in a mathematical language, then thoroughly testing and inspecting it to ensure its correctness. This study examined consensus processes and the role formal approaches play in this space to help with the construction of a trustworthy blockchain-based system. To ensure the accuracy of such procedures, this article discusses the current state of applying formal methods to the blockchain consensus process.
Etrog: Ethereum Based Social Media

Abstract: Today’s OSNs are controlled by major companies such as Facebook, Instagram and Twitter who have complete control over user data. These companies store the user data in a centralized server which.This has several issues. For example, if their servers go down there no backup copies might be available and the chances of getting hacked are more in a centralized architecture. These companies can also censor the contents and post targeted ads in the user feeds. To overcome the drawbacks of the centralized OSNs, in this paper we are implementing a decentralized social media with the help of Ethereum smart contracts. Here, the data is not stored in a centralized server and the major disadvantages of centralized servers can be overcome by this approach.
Database Security with Fingerprint Masking

Abstract: Information in numerous areas including medication, business, and science wasso crucial, in which directories are utilized efficiently for information sharing.Nevertheless, the directories confront the danger to be pirated, taken or even misused, which might lead to many of protection risks regarding ownership rights, informationtampering as well as secrecy safeguards. Watermarking is the technology which is used to secure the database from the various attacks.It is also used to hide the content from the un authorized users. The majority of state-of-the- art techniques alter the initial information to a big level, lead to information quality wreckage,and then can't attain balance that is good between robustness from malicious strikes as well as information restoration. we suggest a reversible and robust watermarking method with Genetic Algorithm and Histogram ShiftingWatermarking (GAHSW) to relation the numerical data. This algorithm used to get the unique key for the database as well as the watermarked place. Experimental evaluation show the usefulness and methods within theterminology of robustness from malicious strikes as well as upkeep of information quality.We have achieved the security level on comparing with existing system
Students’ Perceptions of the Use and Efficacy of Error Correction Symbols on Written Drafts: A Study at the University of Technology and Applied Sciences Ibra, Oman

The present quantitative research study investigates students’ perceptions of the use and efficacy of error correction symbols on written drafts. The study is significant because it has pedagogical implications for teachers, students, and policymakers. It answers the pertinent research question “whether the use of error correction symbols on students’ written drafts is helpful for students in improving their writing accuracy or not”. A structured questionnaire was distributed to students (N=396) via Moodle LMS to collect data. Study participants were General Foundation Program students in the English Language Centre of the University of Technology and Applied Sciences-Ibra, Sultanate of Oman. The findings of this study suggest that the students found the use of error correction symbols on their written drafts practical, helpful, and effective in improving their writing accuracy. Nevertheless, a similar research study needs to be conducted in other higher educational institutions in Oman to sustain the policy directions.
Ability of error correlations to improve the performance of variational quantum algorithms

The quantum approximate optimization algorithm (QAOA) has the potential of providing a useful quantum advantage on noisy intermediate-scale quantum (NISQ) devices. The effects of uncorrelated noise on variational quantum algorithms such as QAOA have been studied intensively. Recent experimental results, however, show that the errors impacting NISQ devices are significantly correlated. We introduce a model for both spatially and temporally (non-Markovian) correlated errors based on classical environmental fluctuators. The model allows for the independent variation of the marginalized spacetime-local error probability and the correlation strength. Using this model, we study the effects of correlated stochastic noise on QAOA. We find evidence that the performance of QAOA improves as the correlation time or correlation length of the noise is increased at fixed local error probabilities. This shows that noise correlations in themselves need not be detrimental for NISQ algorithms such as QAOA.
Semi-Supervised Learning of Optical Flow by Flow Supervisor

A training pipeline for optical flow CNNs consists of a pretraining stage on a synthetic dataset followed by a fine tuning stage on a target dataset. However, obtaining ground truth flows from a target video requires a tremendous effort. This paper proposes a practical fine tuning method to adapt a pretrained model to a target dataset without ground truth flows, which has not been explored extensively. Specifically, we propose a flow supervisor for self-supervision, which consists of parameter separation and a student output connection. This design is aimed at stable convergence and better accuracy over conventional self-supervision methods which are unstable on the fine tuning task. Experimental results show the effectiveness of our method compared to different self-supervision methods for semi-supervised learning. In addition, we achieve meaningful improvements over state-of-the-art optical flow models on Sintel and KITTI benchmarks by exploiting additional unlabeled datasets. Code is available at https://github.com/iwbn/flow-supervisor.
Unimodal vs. Multimodal Siamese Networks for Outfit Completion

The popularity of online fashion shopping continues to grow. The ability to offer an effective recommendation to customers is becoming increasingly important. In this work, we focus on Fashion Outfits Challenge, part of SIGIR 2022 Workshop on eCommerce. The challenge is centered around Fill in the Blank (FITB) task that implies predicting the missing outfit, given an incomplete outfit and a list of candidates. In this paper, we focus on applying siamese networks on the task. More specifically, we explore how combining information from multiple modalities (textual and visual modality) impacts the performance of the model on the task. We evaluate our model on the test split provided by the challenge organizers and the test split with gold assignments that we created during the development phase. We discover that using both visual, and visual and textual data demonstrates promising results on the task. We conclude by suggesting directions for further improvement of our method.
Generative Multiplane Images: Making a 2D GAN 3D-Aware

What is really needed to make an existing 2D GAN 3D-aware? To answer this question, we modify a classical GAN, i.e., StyleGANv2, as little as possible. We find that only two modifications are absolutely necessary: 1) a multiplane image style generator branch which produces a set of alpha maps conditioned on their depth; 2) a pose-conditioned discriminator. We refer to the generated output as a 'generative multiplane image' (GMPI) and emphasize that its renderings are not only high-quality but also guaranteed to be view-consistent, which makes GMPIs different from many prior works. Importantly, the number of alpha maps can be dynamically adjusted and can differ between training and inference, alleviating memory concerns and enabling fast training of GMPIs in less than half a day at a resolution of $1024^2$. Our findings are consistent across three challenging and common high-resolution datasets, including FFHQ, AFHQv2, and MetFaces.
Designing An Illumination-Aware Network for Deep Image Relighting

Lighting is a determining factor in photography that affects the style, expression of emotion, and even quality of images. Creating or finding satisfying lighting conditions, in reality, is laborious and time-consuming, so it is of great value to develop a technology to manipulate illumination in an image as post-processing. Although previous works have explored techniques based on the physical viewpoint for relighting images, extensive supervisions and prior knowledge are necessary to generate reasonable images, restricting the generalization ability of these works. In contrast, we take the viewpoint of image-to-image translation and implicitly merge ideas of the conventional physical viewpoint. In this paper, we present an Illumination-Aware Network (IAN) which follows the guidance from hierarchical sampling to progressively relight a scene from a single image with high efficiency. In addition, an Illumination-Aware Residual Block (IARB) is designed to approximate the physical rendering process and to extract precise descriptors of light sources for further manipulations. We also introduce a depth-guided geometry encoder for acquiring valuable geometry- and structure-related representations once the depth information is available. Experimental results show that our proposed method produces better quantitative and qualitative relighting results than previous state-of-the-art methods. The code and models are publicly available on https://github.com/NK-CS-ZZL/IAN.
Error Compensation Framework for Flow-Guided Video Inpainting

The key to video inpainting is to use correlation information from as many reference frames as possible. Existing flow-based propagation methods split the video synthesis process into multiple steps: flow completion -> pixel propagation -> synthesis. However, there is a significant drawback that the errors in each step continue to accumulate and amplify in the next step. To this end, we propose an Error Compensation Framework for Flow-guided Video Inpainting (ECFVI), which takes advantage of the flow-based method and offsets its weaknesses. We address the weakness with the newly designed flow completion module and the error compensation network that exploits the error guidance map. Our approach greatly improves the temporal consistency and the visual quality of the completed videos. Experimental results show the superior performance of our proposed method with the speed up of x6, compared to the state-of-the-art methods. In addition, we present a new benchmark dataset for evaluation by supplementing the weaknesses of existing test datasets.
Detecting Deepfake by Creating Spatio-Temporal Regularity Disruption

Despite encouraging progress in deepfake detection, generalization to unseen forgery types remains a significant challenge due to the limited forgery clues explored during training. In contrast, we notice a common phenomenon in deepfake: fake video creation inevitably disrupts the statistical regularity in original videos. Inspired by this observation, we propose to boost the generalization of deepfake detection by distinguishing the "regularity disruption" that does not appear in real videos. Specifically, by carefully examining the spatial and temporal properties, we propose to disrupt a real video through a Pseudo-fake Generator and create a wide range of pseudo-fake videos for training. Such practice allows us to achieve deepfake detection without using fake videos and improves the generalization ability in a simple and efficient manner. To jointly capture the spatial and temporal disruptions, we propose a Spatio-Temporal Enhancement block to learn the regularity disruption across space and time on our self-created videos. Through comprehensive experiments, our method exhibits excellent performance on several datasets.
Automated Kantian Ethics: A Faithful Implementation

As we grant artificial intelligence increasing power and independence in contexts like healthcare, policing, and driving, AI faces moral dilemmas but lacks the tools to solve them. Warnings from regulators, philosophers, and computer scientists about the dangers of unethical artificial intelligence have spurred interest in automated ethics-i.e., the development of machines that can perform ethical reasoning. However, prior work in automated ethics rarely engages with philosophical literature. Philosophers have spent centuries debating moral dilemmas so automated ethics will be most nuanced, consistent, and reliable when it draws on philosophical literature. In this paper, I present an implementation of automated Kantian ethics that is faithful to the Kantian philosophical tradition. I formalize Kant's categorical imperative in Dyadic Deontic Logic, implement this formalization in the Isabelle theorem prover, and develop a testing framework to evaluate how well my implementation coheres with expected properties of Kantian ethic. My system is an early step towards philosophically mature ethical AI agents and it can make nuanced judgements in complex ethical dilemmas because it is grounded in philosophical literature. Because I use an interactive theorem prover, my system's judgements are explainable.
One-vs-the-Rest Loss to Focus on Important Samples in Adversarial Training

This paper proposes a new loss function for adversarial training. Since adversarial training has difficulties, e.g., necessity of high model capacity, focusing on important data points by weighting cross-entropy loss has attracted much attention. However, they are vulnerable to sophisticated attacks, e.g., Auto-Attack. This paper experimentally reveals that the cause of their vulnerability is their small margins between logits for the true label and the other labels. Since neural networks classify the data points based on the logits, logit margins should be large enough to avoid flipping the largest logit by the attacks. Importance-aware methods do not increase logit margins of important samples but decrease those of less-important samples compared with cross-entropy loss. To increase logit margins of important samples, we propose switching one-vs-the-rest loss (SOVR), which switches from cross-entropy to one-vs-the-rest loss for important samples that have small logit margins. We prove that one-vs-the-rest loss increases logit margins two times larger than the weighted cross-entropy loss for a simple problem. We experimentally confirm that SOVR increases logit margins of important samples unlike existing methods and achieves better robustness against Auto-Attack than importance-aware methods.
The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing

Machine learning is transforming the video editing industry. Recent advances in computer vision have leveled-up video editing tasks such as intelligent reframing, rotoscoping, color grading, or applying digital makeups. However, most of the solutions have focused on video manipulation and VFX. This work introduces the Anatomy of Video Editing, a dataset, and benchmark, to foster research in AI-assisted video editing. Our benchmark suite focuses on video editing tasks, beyond visual effects, such as automatic footage organization and assisted video assembling. To enable research on these fronts, we annotate more than 1.5M tags, with relevant concepts to cinematography, from 196176 shots sampled from movie scenes. We establish competitive baseline methods and detailed analyses for each of the tasks. We hope our work sparks innovative research towards underexplored areas of AI-assisted video editing.
The Game of Hidden Rules: A New Kind of Benchmark Challenge for Machine Learning

As machine learning (ML) is more tightly woven into society, it is imperative that we better characterize ML's strengths and limitations if we are to employ it responsibly. Existing benchmark environments for ML, such as board and video games, offer well-defined benchmarks for progress, but constituent tasks are often complex, and it is frequently unclear how task characteristics contribute to overall difficulty for the machine learner. Likewise, without a systematic assessment of how task characteristics influence difficulty, it is challenging to draw meaningful connections between performance in different benchmark environments. We introduce a novel benchmark environment that offers an enormous range of ML challenges and enables precise examination of how task elements influence practical difficulty. The tool frames learning tasks as a "board-clearing game," which we call the Game of Hidden Rules (GOHR). The environment comprises an expressive rule language and a captive server environment that can be installed locally. We propose a set of benchmark rule-learning tasks and plan to support a performance leader-board for researchers interested in attempting to learn our rules. GOHR complements existing environments by allowing fine, controlled modifications to tasks, enabling experimenters to better understand how each facet of a given learning task contributes to its practical difficulty for an arbitrary ML algorithm.
The tropical geometry of causal inference for extremes

Extreme value statistics is the max analogue of classical statistics, while tropical geometry is the max analogue of classical geometry. In this paper, we review recent work where insights from tropical geometry were used to develop new, efficient learning algorithms with leading performance on benchmark datasets in extreme value statistics. We give intuition, backed by performances on benchmark datasets, for why and when causal inference for extremes should be employed over classical methods. Finally, we list some open problems at the intersection of causal inference, tropical geometry and deep learning.
Clustering constrained on linear networks

An unsupervised classification method for point events occurring on a network of lines is proposed. The idea relies on the distributional flexibility and practicality of random partition models to discover the clustering structure featuring observations from a particular phenomenon taking place on a given set of edges. By incorporating the spatial effect in the random partition distribution, induced by a Dirichlet process, one is able to control the distance between edges and events, thus leading to an appealing clustering method. A Gibbs sampler algorithm is proposed and evaluated with a sensitivity analysis. The proposal is motivated and illustrated by the analysis of crime and violence patterns in Mexico City.