A Novel Dual of Shannon Information and Weighting Scheme

## Abstract

Abstract Information theory has achieved great success in not only communication technology where it was originally developed for by Shannon but also many other digital fields such as machine learning and artificial intelligence. Inspired by the famous weighting scheme TF-IDF, we discovered that information entropy has a natural dual. We complement the classical Shannon information theory by proposing a novel quantity, namely troenpy. Troenpy measures the certainty, commonness and similarity of the underlying distribution. To demonstrate its usefulness, we propose a troenpy based weighting scheme for document with class labels, namely positive class frequency (PCF). On a collection of public datasets we show the PCF based weighting scheme outperforms the classical TF-IDF and a popular Optimal Transportation based word moving distance algorithm in a kNN setting. We further developed a new odds-ratio type feature, namely Expected Class Information Bias(ECIB), which can be regarded as the expected odds ratio of the information quantity entropy and troenpy. In the experiments we observe that including the new ECIB features and simple binary term features in a simple logistic regression model can further significantly improve the performance. The simple new weighting scheme and ECIB features are very effective and can be computed with linear order complexity.

## 1 Introduction

The classical information theory was originally proposed by Shannon(Shannon, ) to solve the message coding problem in telecommunication. It turned out that it has far more profound impact beyond communication theory, and it has shaped all aspects of our science, engineering and social science by now. The core concept entropy was coined to measure the expected rareness or surpriseness of a random variable $X$ across its distribution. In the literature entropy is usually regarded as the *information*. The mutual information (MI) between two variables is the difference of the entropy of a variable from its conditional entropy given the other variable. MI maximization principle also has been studied and used widely in machine learning. Recently MI has also been employed as part of the objective function for optimization in neural network models based representation learning(Belghazi et al., ; Hjelm et al., ).

Along another line, weighting scheme has been used extensively in information retrieval tasks. Term Frequency-Inverse Document Frequency(TF-IDF), a simple statistic heuristic proposed by (Sparck Jones, ) has been widely used as a weighting method over half a century in information retrieval and natural language processing. It weighs down a term if its document frequency increases in the corpus, as it becomes less effective to distinguish from others when it gets popular and its appearance brings less *surpriseness* in the sense of Shannon self-information. This simple but effective algorithm has achieved great success as a robust weighting scheme. Even today many search engines and digital database systems still employ TF-IDF as an important default algorithm for ranking.

In the past decades a few researchers have intensively investigated on it for a better theoretical understanding of the underlying mechanism rather than a heuristic and intuition argument. Robertson () justified it as an approximate measure of naive Bayes based probability relevance model in information retrieval. Some researchers tried to explain from the information theory point view. (Aizawa, ) interpreted it as some probability weighted amount of information. Siegler and Witbrock () interpreted IDF for a term exactly as the mutual information between a random variable representing a term sampling and a random variable representing a document sampling from a corpus. Many other variants of the term frequency have been proposed in the literature. For example, BM25(Robertson, ) based on probabilistic retrieval framework was further proposed and it has been widely used by search engines to estimate the relevance of documents to a given search query. In general the derived applications go far beyond text processing and information retrieval community.

The connection between TF-IDF and information theory mentioned above is quite motivating. This makes us wonder if there are other simple and effective weighting schemes that can be established from information theory. In order to achieve this goal, it turns out that we first developed a new concept of information quantity, namely *troenpy*, a natural dual to entropy, and then used it to define a new type of weighting scheme which works very well in the extensive experiments as we hoped.

In the following we will first introduce troenpy, the novel dual of Shannon entropy, and share some insights we have for this innovation. Then for the classical task of supervised document classification, we will develop a troenpy based weighting scheme for document representation. This weighting scheme makes use of the documents class label distribution and helps improving the model performance very significantly. Employing both entropy and troenpy, we will also define some new odds-ratio based class bias features leveraging the document class label distribution. Finally evaluating under the simple KNN and logistic regression settings, we show that the proposed new weighting scheme and new features are very effective and achieved substantial error reduction compared with the TF-IDF and a popular optimal transportation based document classification algorithm on a collection of widely used benchmark data sets.

## 2 Dual of Shannon Entropy

We fix the notations first. Here we let $X$ indicate a discrete random variables with probability mass function $p_{X}(x)$. The Shannon entropy (sometimes also called self-information) measures the uncertainty of the underlying variable, or the level of *surpriseness* of an outcome in literature. In
this work we purposely call it Negative Information(NI) for showing the duality nature later. That is,

$$ $\textrm{NI}(x):=-log(p_{X}(x))=log\frac{1}{p_{X}(x)}.$ (1) $$

Now since Shannon information measures *surpriseness*, can we measure the *commonness* instead? This is exactly the contrary to the Shannon information, the dual of Negative Information.

###### Definition 1

We define Positive Information (PI) of an outcome $x$ as

$$ $\textrm{PI}(x):=-log(1-p_{X}(x))=log\frac{1}{1-p_{X}(x)}.$ (2) $$

So PI has the same value range $[0,\infty)$ as NI. Note if we denote $\bar{x}$ the complement of outcome $x$, then $\textrm{PI}(x)=\textrm{NI}(\bar{x})$. For discrete random variables with probabilities $p_{i}$, where $i\in\{1,\dots,K\}$, the value $\frac{1}{1-p_{i}}$ is the measure of *non-surprisenss* or *commonness*.

Naturally by taking expectation across the distribution, we propose a dual quantity of entropy, namely troenpy, to measure the certainty of $X$. Troenpy is simply the distributed *positive* information, while entropy measures the distributed *Negative* Information (NI). Troenpy reflects the level of *reliability* of the $X$ outcomes that the data conceals.

###### Definition 2

The troenpy of a discrete random variable $X$ is defined as the expectation of the PIs,

$$ $T(x):=-\sum_{x}p_{X}(x)log(1-p_{X}(x)).$ (3) $$

For continuous $X$, the differential troenpy is formally defined correspondingly if the integral is finite,

$$ $T(x):=-\int p_{X}(x)log(1-p_{X}(x))dx.$ (4) $$

Note conceptually if the certainty increases, it means some outcomes gain more weight and the disorder or uncertainty of the outcomes decreases correspondingly. Because of the intrinsic nature of troenpy, it naturally serves as a weighting scheme measuring the reliability or certainty of a random variable. More certainty means more predictability. If a random variable has very low certainty, this just means it has a high entropy and is very noisy. Thus it is not a good candidate for prediction purposes and should be correspondingly down-weighted.

Figure: Figure 1: Errors of document classification for 7 Datasets with TF-IDF and TF-PI
Refer to caption: /html/2304.12814/assets/troenpy_fig1_04242023.png

###### Definition 1

###### Definition 2

## 3 Weighting Scheme for Supervised Documents Classification

In this section we first briefly review the information theoretic interpretation of TF-IDF, then naturally we define a new weighting scheme using the newly proposed troenpy as an analogue.

### 3.1 Review of IDF

Here we follow the information theoretic view mentioned above (Aizawa, ). We consider the classical text documents classification task in the routine supervised learning setting. The typical scenario is that given a corpus collection of documents $\text{D}_{1},\dots,\text{D}_{n}$, where n denotes the total number of documents. Each document $\text{D}_{i}$ has a class label $y_{i}$ from a finite class label set $Y=\{1,2,\dots,K\}$, where $K$ is the total number of classes. For a given word term w, let $d$ denote the number of documents where $w$ appears. Then the IDF is simply given by the following:

$$ $IDF(w)=1+log\frac{n}{1+d}$ (5) $$

It can be interpreted as the *self*(negative) information in information theory, which measures the *surprisal* of the term t. The idea follows as below: Fix a word $w$ with document frequency d in a collection of $N$ documents, then the probability of $w$ appear in a document D can be approximated by $Prob(w\in D)=\frac{d}{N}$. Then the negative-information $NI(w)=-logProb(w\in D)=log\frac{N}{d}$. To smooth out the case when $d=0$, adding 1 to the denominator gives $NI(x)\approx log\frac{N}{d+1}$. Also, the summation of all TF-IDFs, each of which represents bits of information weighted by the probability of a term, also recovers the mutual information between terms and documents.

### 3.2 Positive Class Frequency

In this section we will make use of the document class distribution and define a new term weighting method, which can be applied later for the classification task. First for all the $n$ documents in the corpus, we collect the counts of documents for each class. We denote the class label distribution as $C=\{C_{1},\dots,C_{K}\}$, where $C_{i}$ is the count of the $i^{th}$ class label. Normalizing by dividing the total number of documents $n$ gives the probability distribution $\overrightarrow{c}=\{c_{1},\dots,c_{K}\}$, where $c_{i}=\frac{C_{i}}{n}$. This vector $\overrightarrow{c}$ contains the global distribution information and we can define two intrinsic quantities measuring the certainty and uncertainty.

###### Definition 3

We define Positive Class Frequency(PCF) for $C$ as the troenpy of $\overrightarrow{c}$. Similarly, Negative (or Inverse) Class Frequency(NCF or ICF) as the entropy of $\overrightarrow{c}$.

$$ $\begin{split}\textrm{PCF}(C):&=Troenpy(c)\\ \textrm{NCF}(C):&=Entropy(c)\\ \end{split}$ (6) $$

For the whole documents collection (abbreviated as $\textrm{DC}_{*}$), the PCF of the normalized label vector $\overrightarrow{c}$, denoted as $PCF_{*}$ is a constant for each term indicating the certainty level of the whole label distribution at the collection population level. Restricting to the documents with the term $w$ present (abbreviated as $DC_{1}$), the corresponding PCF is denoted as $\textrm{PCF}_{1}$. Similarly, $\textrm{PCF}_{-1}$ denotes the PCF for documents without the term $w$ (abbreviated as $DC_{-1}$). We propose using the difference $\textrm{PCF}_{1}-\textrm{PCF}_{*}$ between $\textrm{PCF}_{1}$ and $\textrm{PCF}_{*}$ as a term weighting reflecting the certainty gain due to the presence of the term $w$. Without abuse of notation, we simply keep using PCF to denote this new weighting scheme. It can also be interpreted as a conditional troenpy condition on the knowledge of the presence or absence of the term $w$. Note in the classical TF-IDF setting and general machine learning literature, such label distribution information has never been made use of before.

To combine the IDF and PCF weightings, we propose using their multiplication $\text{PCF}\cdot\text{IDF}$, abbreviated as PI, as the weighting. Hence multiplying with the term frequency gives the name TF-PI. So in our setting each document can be represented as a vector of word term frequencies multiplied with selected weighting method applied such as $doc_{i}=[tf_{1}\textrm{PI}_{1},\dots,tf_{m}\textrm{PI}_{m}]$, where $m$ is the number of unique selected terms in a document.

On the other hand, the entropy based NCF is not suitable for weighting as they are the negative information measuring the uncertainty. We will illustrate and explain this phenomenon elsewhere Zhang ().

###### Definition 3

## 4 Class Information Bias Features and Binary Term Frequency Features

In this section we introduce two types of features for document representation: the odds ratio based features for class information distribution and a simple binary term frequency feature. For abbreviation, we denote these two features as 2B features in the experiments.

### 4.1 Odds-Ratio based Class Information Bias Features

The idea is that both the TF-IDF and TF-PI are obtained from a term frequency multiplied with a weight information quantity measuring their rareness or certainty, instead we can weight these term frequencies by how biased they distributed across the classes.
This idea was inspired by an algorithm called Delta-IDF. In a simple two class sentiment classification setting, Martineau and Fanin () proposed first taking the difference of the IDFs between the documents of the positive class and the documents of the negative class and then multiplying with the term frequency to give their delta-TFIDF. That is, $tf_{w}[log\frac{P}{P_{w}}-log\frac{N}{N_{w}}]$, where $P$ and $N$ respectively stand for the total numbers of positive documents and negative documents, and the $P_{w}$ and $N_{w}$ respectively stand for the total numbers of positive documents with the term $w$ appears and the total number of negative documents with term $w$ appears. So the difference between the IDFs of the two collections of documents are exactly the odds ratio of the documents counts for the two complementary collections of documents, which can be rewritten as $log\frac{PN_{w}}{P_{w}N}$.

Motivated by the above, we can first compute the NCF and PCF difference for any class $i$, which gives the the Class Information Bias (CIB) features. And then we take the weighted average of such CIB features across all $K$ classes. We call these new features the Expected Class Information Bias (ECIB) features. Specifically for a term $w$, we first use $n_{w}$ denote the number of documents with $w$ present and $n_{iw}$ denote the number of documents with class label $i$ and w present. Then the NCF based CIB for class $i$ is given as $\textrm{CIB}_{i}(w)=log\frac{C_{i}}{1+n_{iw}}-log\frac{n-C_{i}}{1+n_{w}-n_{iw}}$, as $(n-C_{i})$ stands for the total documents not in class $i$ and $(n_{w}-n_{iw})$ stands for the total number of documents not in class $i$ but with $w$ appears. Similarly, the PCF based CIB is given as $log\frac{C_{i}}{1+C_{i}-n_{iw}}-log\frac{n-C_{i}}{1+n-C_{i}-n_{w}+n_{iw}}$.

Therefore for each term $w$, we can define two such distributed Class Information Bias features, one using NCF and one using PCF. The expected CIB features are precisely given by the following.

$$ $\begin{split}&\text{CIB-NCF}(w):\\ &=\sum^{K}_{i=1}\frac{C_{i}}{n}(log\frac{C_{i}}{1+n_{iw}}-log\frac{n-C_{i}}{1+n_{w}-n_{iw}})\\ &\text{CIB-PCF}(w):=\sum^{K}_{i=1}\frac{C_{i}}{n}(log\frac{C_{i}}{1+C_{i}-n_{iw}}\\ &-log\frac{n-C_{i}}{1+n-C_{i}-n_{w}+n_{iw}})\\ \end{split}$ (7) $$

The effect for this ECIB feature is that words that are evenly distributed for their contribution of the information quantities in a class and the rest of the class get little weight, while words that are prominent in some class for their contribution of the information quantities get more weight. So the terms characterizing specific classes are relatively better weighted as they are more representative.

### 4.2 Binary Term Frequency

The binary term frequency (BTF) is simply a binary feature for each term $w$. BTF($w$) is 1 if w is present in a document and it is 0 if it is absent in a document. BTF gives the most naive representation of a document, regardless of frequency counts. We notice that BTF features are actually quite informative and together with TF-IDF can significantly improve the classification performance in the kNN setting. One can achieve this by simply summing the TF-IDF based pairwise document distance and the BTF features based document pairwise distance as the final document pairwise distance.

Figure: Figure 2: Error rates of document classification using 2B features in logistic regression classifier
Refer to caption: /html/2304.12814/assets/troenpy_fig2_04242023.png

## 5 Datasets and Experiment

The goal of our experiments in this section is to validate our proposed weighting schemes and features for the supervised document classification tasks, and compare with the baseline algorithms. To achieve this we include seven text document datasets that are often used for the documents classification tasks in the literature. Three datasets already have a training dataset and a test dataset split while the rest four have no such splits. The experiments of supervised document classification tasks have two settings for the evaluation: a simple kNN setting and a logistic regression setting. The evaluation metric is the error rates on the test datasets.

### 5.1 Datasets

Here we follow closely the setup of Yurochkin et al. ().
We use the popular seven open source datasets below for the study on KNN based classification tasks. Note these datasets have been extensively used numerous times for the classification tasks. The datasets include BBC sports news articles labeled into five sports categories (BBCsports); medical documents labeled into 10 classes of cardiovascular disease types( Ohsumed); Amazon reviews labeled by three categories of Positive, Neutral and Negative (Amazon); tweets labeled by sentiment categories (Twitter); newsgroup articles labeled into 20 categories (20 News group); sentences from science articles labeled by different publishers ( Classic) and Reuters news articles labeled by eight different topics (R8). The more detailed information about the datasets can be found in the references mentioned above. For the datasets with no explicit train and test splits, we use the common 80/20 train-test split and report the performance result based on fifty repeats of random sampling.

To minimize the datasets version mismatch, in all the experiments we use the raw text documents rather than some pre-processed intermediate formats such as some of the processed datasets provided in Kusner et al. ().

Figure: Figure 3: t-SNE on R8 data
Refer to caption: /html/2304.12814/assets/R8_idf_tsne.png

### 5.2 Experiment Settings

For the evaluation of supervised documents classification on term frequencies and their weighting, we include the classical TF-IDF document representation as a baseline. The pairwise document distance in kNN setting is computed using the TF-IDF represented vectors. For comparison purpose and reference, in the experiments we also include the result of a Word Moving Distance (WMD) based algorithm, namely HOFTT proposed by Yurochkin et al. (). It is a hierarchical optimal transportation distance in the topic spaces of documents. The experiment setting of HOFTT is identical to ours.

#### kNN based Classification:

The features include term frequencies only. The goal is to validate the TF-PI weighting and compare with TF-IDF.
The data pre-processing starts with removing the frequent English words in the stop word list, which can be found in the above references. To ease the kNN evaluation part, we fix the number of closest neighborhoods K=7 rather than dynamically selecting the optimal K. We compute the integrated weighting PI as the product of PCF and IDF, and compare with the IDF weighting for each term frequency. Using the TF-PI and TF-IDF, we obtain the bag-of-words vector representation of each document and take their L2 normalization, and then compute the document pairwise distance following the standard kNN procedures. Again our main goal here is to assess if the proposed PCF weighting is effective and can help improve the classical TF-IDF method.

#### Logistic Regression based Classification:

In this setting we simply replace the simple kNN with a standard logistic regression model instead. Here we have two goals to evaluate. First we need to evaluate if the models have performance improvement when the 2B features are included, compared with the models using only the TF-PI features. So we can assess if 2B features are effective for the document classification task. Second we want to evaluate the PCF weighting effect on the ECIB and BTF features both separately and jointly.

Here the data preprocessing is identical to the kNN classification settings above. We mainly consider three types of features in the experiment, namely the TF-PI features, binary term features (BTF) and the ECIB features.

## 6 Results and Discussion

#### kNN based Classification Experiments:

In Figure 1 we can visually observe that the TF-PI based kNN model uniformly outperformed the classical TF-IDF based kNN across all seven datasets and the improvement is quite substantial for most cases with an average overall error reduction $22.9\%$. Noticeably the R8 dataset achieves the most $53.4\%$ error reduction. These uniform improvement can be explained as the PCF weighting does effectively leverage the certainty and common similarity of class label distributions across the corpus at a term level. For a term, the more PCF it has the better prediction capacity it has. For example, the different news groups in Ng20 actually share many non-stop words in common and some groups are very relevant. The learned similarity information about one group is helpful at predicting a relevant group. We also observe only slight improvement on the Twitter and BBC sport datasets which might be simply due to the small sample sizes. The Twitter has 3115 samples and BBCsport has only 737 samples, which are quite small compared with other datasets. Additionally, the Twitter sentiment dataset has three class labels consisting of positive, neutral and negative. The extreme polarity of the classes is often consistent with the fact that relatively less common description words are shared across the classes.

#### t-SNE:

We also use the popular t-SNE by van der Maaten and Hinton () to visualize the TF-IDF and TF-PI classification effect on the R8 dataset. In Figure 3, the TF-PI appears to cluster relatively closer for each class labels and clusters are relatively separated from other cluster groups.

#### Word Moving Distance Methods:

In the experiments a modern Optimal Transportation (OT) based Word Moving Distance (WMD) approach HOFTT performs poorly compared with the TF-PI weighting on all dataset except on R8 dataset, on which it is also outperformed by TF-PI employing the additional 2B features. However we are also aware another advanced WMD method Wasserstein-Fisher-Rao(WFR) developed by Wang et al. (), which uses the Fisher-Rao metric for the unbalanced optimal transportation problem. The reported result of WFR is comparable to our proposed methods across the datasets. Unfortunately there are some version mismatch for some datasets as well as slightly different sampling procedure for datasets with no pre-specified train-test splits, so we did not include the corresponding result in our figures. Note also that the general Sinkhorn based algorithms for such OT optimization problems have relatively high computational complexity and so they are quite expensive on computational cost. While the proposed weighting scheme and ECIB features can be obtained in a single scan of the data and the time complexity is linear, they are fast and a lot cheaper on computational cost.

#### Logistic Regression based Experiments:

In Figure 1 we observed the following: (1) for the same TF-PI feature set, the logistic regression model uniformly outperforms the kNN approach across all datasets. This is not surprised as the logistic regression optimizes the term coefficients for optimal fitting the data while the kNN is rigid as given. (2) adding the 2B features of binary term frequency (BTF) and expected class information bias (ECIB) further significantly reduces the errors on most datasets. For the BBC dataset we observed a relatively large error increase, and we hypothesize that this may be due to the very small test sample size of the dataset.

In Figure 2 we reported the results of using BTF and ECIB features in the logistic regression setting. We observed the following. Both BTF and ECIB features are quite effective when used individually alone. ECIB performs better than BTF on all datasets except on the dataset of 20 Newsgroup, where they are relatively close. Simply combining the two features together not necessarily always improves the performance, instead it leads to slightly more errors on a couple of the datasets. We also observe that applying the PCF weighting helps on majority of the cases. Visually the left three bars of light color represent 2B features without PCF weighting while the right three bars of darker color represent corresponding features with PCF weighting applied.

## 7 Conclusion

We proposed the new concept troenpy as the dual of the Shannon entropy for a distribution, which summarizes the certainty or common similarities of the outcome distribution. We also developed a troenpy based simple weighting scheme PCF for documents with labels, which can leverage the document class distribution information across a corpus. Combining the PCF and the well-known TF-IDF gives a new term frequency based document representation. For the document classification task, this TF-PI weighting method significantly outperformed not only the classical TF-IDF weighting method in a fixed KNN setting but also an advanced optimal transportation based word moving distance method on a collection of widely used datasets.

Additionally we also proposed two new features: binary term presence(BTF) feature and expected class information bias (ECIB) feature. The ECIB features are the weighted odds-ratios of the information quantities across all classes, so they weigh the terms according to how much distance they are from the evenly contribution to the two information measures. In a simple logistic regression setting we demonstrated that both features are quite effective when used alone for the document classification task while their combination together with the TF-PI features performs the best on the datasets with further significant error reduction. The proposed algorithms have only linear complexity and run fast. They can be applied to extract useful information from the data with only single scan of the data and then be easily integrated as weightings into many other machine learning tasks.

## Limitations

The current work proposed a new weighting scheme leveraging the document label information and demonstrated the effectiveness on a popular benchmark collection of English datasets. For documents without label information available, the current proposal cannot apply directly. However, these unsupervised tasks often can be reformulated into popular self-supervised problems. And then we can apply the developed methods to process. A complete demonstration and explanation is treated elsewhere Zhang ().

For image processing with pixels values in the typical range [0,255] or other continuous data features, straight applying the above weighting schemes does not work. More methods are needed to be developed for problems in such scenarios.
It will be interesting to check the performance of the proposed algorithms on some other large scale corpus of data-sets and also data-sets in other languages. The weighing scheme application are mainly dealt with the discrete tokens scenarios and it is not clear how these methods can extend to the modern distributed representation scenarios where each token is encoded as a high dimensional vector or tensor in general. Especially the pretrained word vectors or transformer based models are dominating the NLP community, it is not clear for the weighting methods how to integrate with these trends.

## Ethics Statement

The conducted research tries to find principled ways for natural language computing tasks for the general goods and technology advancement of the society.

## Acknowledgments

AJZ thanks Timothy J. Hazen for first introducing TF-IDF to him in 2013. AJZ thanks an anonymous reviewer for several tips on improving the manuscript presentation as well as the inspiring question of how applying troenpy to unlabeled data. The related USPTO Patent 18161067 is pending.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 1
presentation: 2
questions: (1) Shannon entropy acts as a lower bound for the average code length for source entropy. What is the significance of "troenpy" here?
(2) Can you show how "trophy" has significance in asymptotic equipartition property? One can show that given a discrete-time stationary ergodic stochastic process X, - 1/n log(X1, X2,...,Xn) converges to H(X), almost surely. Can you establish such results with"troenpy"?
(3) What is the so-called "dual" of Kullback Leibler divergence? Will that divergence have any role as a "distance measure" of probability measures, and what happens to Pinsker Inequality?
(4) Maximum entropy plays an important role in machine learning and image reconstruction. Where does "troenpy" fit in here?  Does the maximum entropy distributions become "minimum troenpy distributions"?
(5) Does Shannon entropy play an important role in the characterization of typical sets? Where does "troenpy" fit in here?
soundness: 1
strengths: Apart from the enthusiasm of the authors to conceive the idea and write this paper, I do not see any major strengths in this paper.
summary: This paper introduces the so-called  "troenpy" which authors call dual. But I doubt whether this is a proper terminology. This measure is applied to a weighting scheme for supervised document classification.  Simple mathematical properties are presented as Theorems without having any theoretical results. Apart from this, the paper only makes negligible contributions and completely ignores the fundamentals of information theory. Experiments are very limited, and events will not be enough for undergraduate assignments. Most of the paper is filled with trivial calculations.
weaknesses: This paper completely ignores the various results in information theory, coding theory, and machine learning, where Shannon entropy plays an important role (see my questions below). Apart from some trivial calculations, the paper has no theoretical contributions. The motivations or mathematical significance of so-called measures of certainty are not very apparent. There were some classical generalizations of entropy as Renyi entropy exists in the literature, and though authors call this some kind of dual, no connections to existing literature were made. One can still appreciate the results without any mathematical backing, but this paper does not show any extensive practical applications--that too considering that Shannon entropy appears in so many applications from maximum entropy methods to reinforcement learning. Please see my question below.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: 1- Given that entropy has broad applications across various fields, could you clarify whether troenpy has potential uses beyond document classification? The current scope is quite narrow, and identifying additional applications could help to better demonstrate its value and versatility.

2-  Does troenpy come with any theoretical guarantees that it is the optimal measure for assessing certainty? Could you clarify whether alternative formulations, such as using the logarithm of another decreasing function of p(x), might also be valid? Providing insights into the theoretical underpinnings of troenpy’s formulation would enhance its credibility.

3- Is the primary goal of your research to derive a new measure (troenpy) or to improve upon TF-IDF? If the intention is to present troenpy as a novel measure, it would be helpful to discuss potential applications beyond document classification, as this focus currently feels quite limited. Conversely, if the emphasis is on document classification, the paper leans toward being empirical in nature. It appears there is a mix of these two focuses, which makes the methodological framework somewhat unclear. Could you clarify your primary aim and the intended contributions of the paper?
soundness: 3
strengths: 1- The paper is well written and quite easy to follow 

2- The experimental seems to show the efficiency of the proposed method.
summary: The paper presents a new metric called "troenpy," which is designed to complement Shannon entropy. While entropy measures uncertainty, troenpy aims to quantify certainty. Additionnally, the authors suggest a new weighting method for documents with class labels, called positive class frequency (PCF). They demonstrate that this method significantly outperforms other existing methods.
weaknesses: 1.The paper currently lacks a rigorous theoretical foundation justifying the choice of log(1−p(x)) as the basis for troenpy. While some intuitive motivations are provided, they don’t fully explain why this particular transformation should be optimal or preferable over other possible functions, such as log(g(p(x))). To strengthen the paper, it would be helpful if you could provide a more formal theoretical analysis or justification for the choice of log(1−p(x)) over other alternatives. This could involve deriving troenpy from first principles in a way that demonstrates its uniqueness or optimality for measuring certainty. Such an approach would make troenpy more compelling by showing that this transformation is not only intuitively sound but theoretically motivated as well.

2.The paper’s methodological presentation is somewhat unclear in terms of focus. While troenpy is presented as the main contribution, it is ultimately used only as part of the Positive Class Frequency (PCF) weighting scheme, rather than being explored on its own. At the same time, other features, such as Class Information Bias and Binary Term Frequency, are introduced without a clear link to troenpy, which can dilute the focus of the contribution. To enhance clarity, it may help to more clearly separate the different contributions or to highlight how they relate to the central concept of troenpy. If troenpy is indeed the primary contribution, consider focusing on its properties and potential applications more directly, perhaps by exploring it in different contexts out of document classification or directly comparing it with other certainty measures. Alternatively, if the emphasis is on the PCF and the broader feature set for document classification, reframing troenpy as one component of a larger methodological toolkit could provide a more cmprehensive narrative.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 015: Troenpy is not a "dual" in a precise mathematical sense. It is an interesting and novel variant.

053: This is a very incomplete summary of theoretical justifications for TFIDF. See also, among others, https://link.springer.com/chapter/10.1007/11575832_33, Deriving TF-IDF as a Fisher Kernel.

072: Cite and compare experimentally to previous methods that use odds ratios. See https://dl.acm.org/doi/abs/10.1145/1458082.1458119 "BNS feature scaling: an improved representation over tf-idf for svm text classification" and https://www.jmlr.org/papers/volume3/forman03a/forman03a_full.pdf "An Extensive Empirical Study of Feature Selection Metrics for Text Classification.

114: The new concept, troenpy, is similar to the so-called "one versus all" approach to multiclass classification (which more precisely should be called "one versus rest").

119: "It turns out that the integral is zero." This sentence seems out of context and unsupported. The whole paragraph 116 to 124 seems incomplete and not needed.

203: It use confusing to use the word "frequency" for a value such as d that is an integer count. Use the word "count" as on line 214.

285: The word "negentropy" may appear in the 1944 book by Schrodinger, but presumably not as a formal measure, since the book predates Shannon. Note that the reference is formatted incorrectly, since Schrödinger is the author's family name, not Erwin.

357: Binary term frequency is a standard approach for document classification. See http://kamalnigam.com/papers/multinomial-aaaiws98.pdf, "A Comparison of Event Models for Naive Bayes Text Classification."

403: Does the absence of a validation set mean that there is absolutely no hyperparameter optimization?

429: It seems that all methods use raw TF, i.e., raw word counts. It is well-known that log(TF) or some other squashing function of TF usually gives better accuracy. The reason is burstiness, as discussed in the paper mentioned above, "Deriving TF-IDF as a Fisher Kernel."

490: It is not good enough to just say that previous work used different datasets. Get or implement Wang's method and run it on your datasets.

502: The meaning of "adding" is not clear.

506: Do additional experiments to explain the error increase; do not merely speculate.
soundness: 3
strengths: Entropy of the complement probabilities has not been studied before in the research literature, and makes sense as a useful measure. The experimental results are believable.
summary: Note added on December 3: I am glad that the authors found my review to be useful. I agree with them that the other review with a score of 1 is too extreme, in its score and its wording. However, the authors need to understand better the limitations of their submission, and what is required by a selective venue such as ICLR.

"We did have other successful applications such as the large language models, recommendation system and computer vision etc, which we treated elsewhere as a series of following work" is not a reason to accept the current paper. Acceptance should be based only on actual contributions, not on unsubstantiated claims or promises.

The experimental section of the submission is quite weak, as pointed out previously. The authors did not cite, and did not compare to, the state-of-the-art in the family of TFIDF-based methods for document classification. The BNS method is just one of these methods. It is the duty of the authors to do a careful literature search, and to convince reviewers that the submission includes fair comparisons with the previous best related methods. The submission does not succeed currently in this direction.

=============

This paper introduces a variation of entropy that can be called entropy of the complement probabilities, $-\sum_i p_i \log(1 - p_i)$. The paper uses this as a weighting approach for words in document classification, and gets better accuracy than using alternative weighting methods.
weaknesses: Section 3 and the experiments based on it are clear and persuasive. Section 4 is less clear.

The experimental work is actually very limited, using only seven small datasets and essentially copying the experimental design of just one previous paper.

Although this submission is interesting, it is about document classification based on bag of words, which is mostly obsolete given the availability of large language models which understand document semantics, and hence can classify text with much better accuracy.

Meta-review: This paper considers a dual of Shannon entropy. Unfortunately reviewers have major concerns with the contributions; please revise accordingly and I encourage a stronger resubmission as there are some good ideas in this paper. While reviews were mixed, there is a lot of context from information theory missing in the presentation here

Decision: Reject