OpenRaiser/Intern-Atlas huggingface dataset:
Fast Online EM for Big Topic Modeling
Venue: arXiv cs.LG 2015
Year: 2015

The expectation-maximization (EM) algorithm can compute the maximum-likelihood (ML) or maximum a posterior (MAP) point estimate of the mixture models or latent variable models such as latent Dirichlet allocation (LDA), which has been one of the most popular probabilistic topic modeling methods in the past decade. However, batch EM has high time and space complexities to learn big LDA models from big data streams. In this paper, we present a fast online EM (FOEM) algorithm that infers the topic distribution from the previously unseen documents incrementally with constant memory requirements. Within the stochastic approximation framework, we show that FOEM can converge to the local stationary point of the LDA's likelihood function. By dynamic scheduling for the fast speed and parameter streaming for the low memory usage, FOEM is more efficient for some lifelong topic modeling tasks than the state-of-the-art online LDA algorithms to handle both big data and big models (aka, big topic modeling) on just a PC.
Computing the Gromov hyperbolicity of a discrete metric space
Venue: arXiv cs.CG 2015
Year: 2015

We give exact and approximation algorithms for computing the Gromov hyperbolicity of an n-point discrete metric space. We observe that computing the Gromov hyperbolicity from a fixed base-point reduces to a (max,min) matrix product. Hence, using the (max,min) matrix product algorithm by Duan and Pettie, the fixed base-point hyperbolicity can be determined in O(n^2.69) time. It follows that the Gromov hyperbolicity can be computed in O(n^3.69) time, and a 2-approximation can be found in O(n^2.69) time. We also give a (2 log_2 n)-approximation algorithm that runs in O(n^2) time, based on a tree-metric embedding by Gromov. We also show that hyperbolicity at a fixed base-point cannot be computed in O(n^2.05) time, unless there exists a faster algorithm for (max,min) matrix multiplication than currently known.
Reducing the rank of a matroid
Venue: arXiv cs.DS 2015
Year: 2015

We consider the rank reduction problem for matroids: Given a matroid M and an integer k, find a minimum size subset of elements of M whose removal reduces the rank of M by at least k. When M is a graphical matroid this problem is the minimum k-cut problem, which admits a 2-approximation algorithm. In this paper we show that the rank reduction problem for transversal matroids is essentially at least as hard to approximate as the densest k-subgraph problem. We also prove that, while the problem is easily solvable in polynomial time for partition matroids, it is NP-hard when considering the intersection of two partition matroids. Our proof shows, in particular, that the maximum vertex cover problem is NP-hard on bipartite graphs, which answers an open problem of B. Simeone.
Dynamic Network Control for Confidential Multi-hop Communications
Venue: arXiv cs.IT 2015
Year: 2015

We consider the problem of resource allocation and control of multihop networks in which multiple source-destination pairs communicate confidential messages, to be kept confidential from the intermediate nodes. We pose the problem as that of network utility maximization, into which confidentiality is incorporated as an additional quality of service constraint. We develop a simple, and yet provably optimal dynamic control algorithm that combines flow control, routing and end-to-end secrecy-encoding. In order to achieve confidentiality, our scheme exploits multipath diversity and temporal diversity due to channel variability. Our end-to-end dynamic encoding scheme encodes confidential messages across multiple packets, to be combined at the ultimate destination for recovery. We first develop an optimal dynamic policy for the case in which the number of blocks across which secrecy encoding is performed is asymptotically large. Next, we consider encoding across a finite number of packets, which eliminates the possibility of achieving perfect secrecy. For this case, we develop a dynamic policy to choose the encoding rates for each message, based on the instantaneous channel state information, queue states and secrecy outage requirements. By numerical analysis, we observe that the proposed scheme approaches the optimal rates asymptotically with increasing block size. Finally, we address the consequences of practical implementation issues such as infrequent queue updates and de-centralized scheduling. We demonstrate the efficacy of our policies by numerical studies under various network conditions.
Apollonian Circumcircles of IFS Fractals
Venue: arXiv cs.CG 2015
Year: 2015

Euclidean triangles and IFS fractals seem to be disparate geometrical concepts, unless we consider the Sierpiński gasket, which is a self-similar collection of triangles. The "circumcircle" hints at a direct link, as it can be derived for three-map IFS fractals in general, defined in an Apollonian manner. Following this path, one may discover a broader relationship between polygons and IFS fractals.
A Distance-based Paraconsistent Semantics for DL-Lite
Venue: arXiv cs.AI 2015
Year: 2015

DL-Lite is an important family of description logics. Recently, there is an increasing interest in handling inconsistency in DL-Lite as the constraint imposed by a TBox can be easily violated by assertions in ABox in DL-Lite. In this paper, we present a distance-based paraconsistent semantics based on the notion of feature in DL-Lite, which provides a novel way to rationally draw meaningful conclusions even from an inconsistent knowledge base. Finally, we investigate several important logical properties of this entailment relation based on the new semantics and show its promising advantages in non-monotonic reasoning for DL-Lite.
An Experimental Comparison of Several Clustering and Initialization Methods
Venue: arXiv cs.LG 2015
Year: 2015

We examine methods for clustering in high dimensions. In the first part of the paper, we perform an experimental comparison between three batch clustering algorithms: the Expectation-Maximization (EM) algorithm, a winner take all version of the EM algorithm reminiscent of the K-means algorithm, and model-based hierarchical agglomerative clustering. We learn naive-Bayes models with a hidden root node, using high-dimensional discrete-variable data sets (both real and synthetic). We find that the EM algorithm significantly outperforms the other methods, and proceed to investigate the effect of various initialization schemes on the final solution produced by the EM algorithm. The initializations that we consider are (1) parameters sampled from an uninformative prior, (2) random perturbations of the marginal distribution of the data, and (3) the output of hierarchical agglomerative clustering. Although the methods are substantially different, they lead to learned models that are strikingly similar in quality.
Learning Mixtures of DAG Models
Venue: arXiv cs.LG 2015
Year: 2015

We describe computationally efficient methods for learning mixtures in which each component is a directed acyclic graphical model (mixtures of DAGs or MDAGs). We argue that simple search-and-score algorithms are infeasible for a variety of problems, and introduce a feasible approach in which parameter and structure search is interleaved and expected data is treated as real data. Our approach can be viewed as a combination of (1) the Cheeseman--Stutz asymptotic approximation for model posterior probability and (2) the Expectation--Maximization algorithm. We evaluate our procedure for selecting among MDAGs on synthetic and real examples.
A Bayesian Approach to Learning Bayesian Networks with Local Structure
Venue: arXiv cs.LG 2015
Year: 2015

Recently several researchers have investigated techniques for using data to learn Bayesian networks containing compact representations for the conditional probability distributions (CPDs) stored at each node. The majority of this work has concentrated on using decision-tree representations for the CPDs. In addition, researchers typically apply non-Bayesian (or asymptotically Bayesian) scoring functions such as MDL to evaluate the goodness-of-fit of networks to the data. In this paper we investigate a Bayesian approach to learning Bayesian networks that contain the more general decision-graph representations of the CPDs. First, we describe how to evaluate the posterior probability that is, the Bayesian score of such a network, given a database of observed cases. Second, we describe various search spaces that can be used, in conjunction with a scoring function and a search procedure, to identify one or more high-scoring networks. Finally, we present an experimental evaluation of the search spaces, using a greedy algorithm and a Bayesian scoring function.
Efficient Approximations for the Marginal Likelihood of Incomplete Data Given a Bayesian Network
Venue: arXiv cs.LG 2015
Year: 2015

We discuss Bayesian methods for learning Bayesian networks when data sets are incomplete. In particular, we examine asymptotic approximations for the marginal likelihood of incomplete data given a Bayesian network. We consider the Laplace approximation and the less accurate but more efficient BIC/MDL approximation. We also consider approximations proposed by Draper (1993) and Cheeseman and Stutz (1995). These approximations are as efficient as BIC/MDL, but their accuracy has not been studied in any depth. We compare the accuracy of these approximations under the assumption that the Laplace approximation is the most accurate. In experiments using synthetic data generated from discrete naive-Bayes models having a hidden root node, we find that the CS measure is the most accurate.
FO Model Checking of Interval Graphs
Venue: arXiv cs.DM 2015
Year: 2015

We study the computational complexity of the FO model checking problem on interval graphs, i.e., intersection graphs of intervals on the real line. The main positive result is that FO model checking and successor-invariant FO model checking can be solved in time O(n log n) for n-vertex interval graphs with representations containing only intervals with lengths from a prescribed finite set. We complement this result by showing that the same is not true if the lengths are restricted to any set that is dense in an open subset, e.g., in the set $(1, 1 + \varepsilon)$.
Spectrum Bandit Optimization
Venue: arXiv cs.LG 2015
Year: 2015

We consider the problem of allocating radio channels to links in a wireless network. Links interact through interference, modelled as a conflict graph (i.e., two interfering links cannot be simultaneously active on the same channel). We aim at identifying the channel allocation maximizing the total network throughput over a finite time horizon. Should we know the average radio conditions on each channel and on each link, an optimal allocation would be obtained by solving an Integer Linear Program (ILP). When radio conditions are unknown a priori, we look for a sequential channel allocation policy that converges to the optimal allocation while minimizing on the way the throughput loss or {\it regret} due to the need for exploring sub-optimal allocations. We formulate this problem as a generic linear bandit problem, and analyze it first in a stochastic setting where radio conditions are driven by a stationary stochastic process, and then in an adversarial setting where radio conditions can evolve arbitrarily. We provide new algorithms in both settings and derive upper bounds on their regrets.
Adjacent-Channel Interference in Frequency-Hopping Ad Hoc Networks
Venue: arXiv cs.IT 2015
Year: 2015

This paper considers ad hoc networks that use the combination of coded continuous-phase frequency-shift keying (CPFSK) and frequency-hopping multiple access. Although CPFSK has a compact spectrum, some of the signal power inevitably splatters into adjacent frequency channels, thereby causing adjacent-channel interference (ACI). The amount of ACI is controlled by setting the fractional in-band power; i.e., the fraction of the signal power that lies within the band of each frequency channel. While this quantity is often selected arbitrarily, a tradeoff is involved in the choice. This paper presents a new analysis of frequency-hopping ad hoc networks that carefully incorporates the effect of ACI. The analysis accounts for the shadowing, Nakagami fading, CPFSK modulation index, code rate, number of frequency channels, fractional in-band power, and spatial distribution of the interfering mobiles. Expressions are presented for both outage probability and transmission capacity. With the objective of maximizing the transmission capacity, the optimal fractional in-band power that should be contained in each frequency channel is identified.
On Cooperation in Multi-Terminal Computation and Rate Distortion
Venue: arXiv cs.IT 2015
Year: 2015

A receiver wants to compute a function of two correlated sources separately observed by two transmitters. One of the transmitters may send a possibly private message to the other transmitter in a cooperation phase before both transmitters communicate to the receiver. For this network configuration this paper investigates both a function computation setup, wherein the receiver wants to compute a given function of the sources exactly, and a rate distortion setup, wherein the receiver wants to compute a given function within some distortion. For the function computation setup, a general inner bound to the rate region is established and shown to be tight in a number of cases: partially invertible functions, full cooperation between transmitters, one-round point-to-point communication, two-round point-to-point communication, and the cascade setup where the transmitters and the receiver are aligned. In particular it is shown that the ratio of the total number of transmitted bits without cooperation and the total number of transmitted bits with cooperation can be arbitrarily large. Furthermore, one bit of cooperation suffices to arbitrarily reduce the amount of information both transmitters need to convey to the receiver. For the rate distortion version, an inner bound to the rate region is exhibited which always includes, and sometimes strictly, the convex hull of Kaspi-Berger's related inner bounds. The strict inclusion is shown via two examples.
Causal Independence for Knowledge Acquisition and Inference
Venue: arXiv cs.AI 2015
Year: 2015

I introduce a temporal belief-network representation of causal independence that a knowledge engineer can use to elicit probabilistic models. Like the current, atemporal belief-network representation of causal independence, the new representation makes knowledge acquisition tractable. Unlike the atemproal representation, however, the temporal representation can simplify inference, and does not require the use of unobservable variables. The representation is less general than is the atemporal representation, but appears to be useful for many practical applications.
String Submodular Functions with Curvature Constraints
Venue: arXiv cs.DS 2015
Year: 2015

The problem of objectively choosing a string of actions to optimize an objective function that is string submodular has been considered in [1]. There it is shown that the greedy strategy, consisting of a string of actions that only locally maximizes the step-wise gain in the objective function achieves at least a (1-e^{-1})-approximation to the optimal strategy. This paper improves this approximation by introducing additional constraints on curvatures, namely, total backward curvature, total forward curvature, and elemental forward curvature. We show that if the objective function has total backward curvature σ, then the greedy strategy achieves at least a \frac{1}σ(1-e^{-σ})-approximation of the optimal strategy. If the objective function has total forward curvature ε, then the greedy strategy achieves at least a (1-ε)-approximation of the optimal strategy. Moreover, we consider a generalization of the diminishing-return property by defining the elemental forward curvature. We also consider the problem of maximizing the objective function subject to general a string-matroid constraint. We investigate an applications of string submodular functions with curvature constraints.
On the Entropy of Couplings
Venue: arXiv cs.IT 2015
Year: 2015

In this paper, some general properties of Shannon information measures are investigated over sets of probability distributions with restricted marginals. Certain optimization problems associated with these functionals are shown to be NP-hard, and their special cases are found to be essentially information-theoretic restatements of well-known computational problems, such as the SUBSET SUM and the 3-PARTITION. The notion of minimum entropy coupling is introduced and its relevance is demonstrated in information-theoretic, computational, and statistical contexts. Finally, a family of pseudometrics (on the space of discrete probability distributions) defined by these couplings is studied, in particular their relation to the total variation distance, and a new characterization of the conditional entropy is given.
An Approximate Nonmyopic Computation for Value of Information
Venue: arXiv cs.AI 2015
Year: 2015

Value-of-information analyses provide a straightforward means for selecting the best next observation to make, and for determining whether it is better to gather additional information or to act immediately. Determining the next best test to perform, given a state of uncertainty about the world, requires a consideration of the value of making all possible sequences of observations. In practice, decision analysts and expert-system designers have avoided the intractability of exact computation of the value of information by relying on a myopic approximation. Myopic analyses are based on the assumption that only one additional test will be performed, even when there is an opportunity to make a large number of observations. We present a nonmyopic approximation for value of information that bypasses the traditional myopic analyses by exploiting the statistical properties of large samples.
Separable and transitive graphoids
Venue: arXiv cs.AI 2015
Year: 2015

We examine three probabilistic formulations of the sentence a and b are totally unrelated with respect to a given set of variables U. First, two variables a and b are totally independent if they are independent given any value of any subset of the variables in U. Second, two variables are totally uncoupled if U can be partitioned into two marginally independent sets containing a and b respectively. Third, two variables are totally disconnected if the corresponding nodes are disconnected in every belief network representation. We explore the relationship between these three formulations of unrelatedness and explain their relevance to the process of acquiring probabilistic knowledge from human experts.
Optimal Kullback-Leibler Aggregation via Information Bottleneck
Venue: arXiv cs.SY 2015
Year: 2015

In this paper, we present a method for reducing a regular, discrete-time Markov chain (DTMC) to another DTMC with a given, typically much smaller number of states. The cost of reduction is defined as the Kullback-Leibler divergence rate between a projection of the original process through a partition function and a DTMC on the correspondingly partitioned state space. Finding the reduced model with minimal cost is computationally expensive, as it requires an exhaustive search among all state space partitions, and an exact evaluation of the reduction cost for each candidate partition. Our approach deals with the latter problem by minimizing an upper bound on the reduction cost instead of minimizing the exact cost; The proposed upper bound is easy to compute and it is tight if the original chain is lumpable with respect to the partition. Then, we express the problem in the form of information bottleneck optimization, and propose using the agglomerative information bottleneck algorithm for searching a sub-optimal partition greedily, rather than exhaustively. The theory is illustrated with examples and one application scenario in the context of modeling bio-molecular interactions.
The finite index basis property
Venue: arXiv cs.DM 2015
Year: 2015

We describe in this paper a connection between bifix codes, symbolic dynamical systems and free groups. This is in the spirit of the connection established previously for the symbolic systems corresponding to Sturmian words. We introduce a class of sets of factors of an infinite word with linear factor complexity containing Sturmian sets and regular interval exchange sets, namemly the class of tree sets. We prove as a main result that for a uniformly recurrent tree set $F$, a finite bifix code $X$ on the alphabet $A$ is $F$-maximal of $F$-degree $d$ if and only if it is the basis of a subgroup of index $d$ of the free group on $A$.
Towards a better compressed sensing
Venue: arXiv cs.IT 2015
Year: 2015

In this paper we look at a well known linear inverse problem that is one of the mathematical cornerstones of the compressed sensing field. In seminal works \cite{CRT,DOnoho06CS} $\ell_1$ optimization and its success when used for recovering sparse solutions of linear inverse problems was considered. Moreover, \cite{CRT,DOnoho06CS} established for the first time in a statistical context that an unknown vector of linear sparsity can be recovered as a known existing solution of an under-determined linear system through $\ell_1$ optimization. In \cite{DonohoPol,DonohoUnsigned} (and later in \cite{StojnicCSetam09,StojnicUpper10}) the precise values of the linear proportionality were established as well. While the typical $\ell_1$ optimization behavior has been essentially settled through the work of \cite{DonohoPol,DonohoUnsigned,StojnicCSetam09,StojnicUpper10}, we in this paper look at possible upgrades of $\ell_1$ optimization. Namely, we look at a couple of algorithms that turn out to be capable of recovering a substantially higher sparsity than the $\ell_1$. However, these algorithms assume a bit of "feedback" to be able to work at full strength. This in turn then translates the original problem of improving upon $\ell_1$ to designing algorithms that would be able to provide output needed to feed the $\ell_1$ upgrades considered in this papers.
The Evolution of Beliefs over Signed Social Networks
Venue: arXiv cs.SI 2015
Year: 2015

We study the evolution of opinions (or beliefs) over a social network modeled as a signed graph. The sign attached to an edge in this graph characterizes whether the corresponding individuals or end nodes are friends (positive links) or enemies (negative links). Pairs of nodes are randomly selected to interact over time, and when two nodes interact, each of them updates its opinion based on the opinion of the other node and the sign of the corresponding link. This model generalizes DeGroot model to account for negative links: when two enemies interact, their opinions go in opposite directions. We provide conditions for convergence and divergence in expectation, in mean-square, and in almost sure sense, and exhibit phase transition phenomena for these notions of convergence depending on the parameters of the opinion update model and on the structure of the underlying graph. We establish a {\it no-survivor} theorem, stating that the difference in opinions of any two nodes diverges whenever opinions in the network diverge as a whole. We also prove a {\it live-or-die} lemma, indicating that almost surely, the opinions either converge to an agreement or diverge. Finally, we extend our analysis to cases where opinions have hard lower and upper limits. In these cases, we study when and how opinions may become asymptotically clustered to the belief boundaries, and highlight the crucial influence of (strong or weak) structural balance of the underlying network on this clustering phenomenon.