Tsetlin Machine for Solving Contextual Bandit Problems

Abstract: This paper introduces an interpretable contextual bandit algorithm using Tsetlin Machines, which solves complex pattern recognition tasks using  propositional (Boolean) logic. The proposed bandit learning algorithm relies on straightforward bit manipulation, thus simplifying computation and interpretation. We then present a mechanism for performing Thompson sampling with Tsetlin Machine, given its non-parametric nature. Our empirical analysis shows that Tsetlin Machine as a base contextual bandit learner outperforms other popular base learners on eight out of nine datasets. We further analyze the interpretability of our learner, investigating how arms are selected based on propositional expressions that model the context.

Reviews and author responses:

 Thanks for the responses. I keep my score. 

 Thank you for the response. I have updated the score. It will be great to add more details of TM, in particular, how it is compatible with different types of data.

 We thank the reviewer for the valuable feedback.

The complexity of the algorithm is proportional (linear) to the number of features (input bits). E.g., doubling the number of features doubles the number of computations (https://ieeexplore.ieee.org/document/9316190, Sect II.E). The accuracy depends on the encoding scheme, e.g. explored for binning and logical booleanization in: 

https://www.frontiersin.org/articles/10.3389/fcteg.2021.778118/full

 We thank the reviewer for their feedback. 
1. Theoretical results and convergence of the learning procedure for TM is provided in [2,3]. How to extend these results to TM for the contextual bandit problem is an open research question. We here provide an empirical evaluation, like done for Thompson Sampling when introduced at NeurIPS in 2011 (https://proceedings.neurips.cc/paper/2011/file/e53a0a2978c28872a4505bdb51db06dc-Paper.pdf). Only later was the theoretical foundation developed in depth. The primary motivation of using TM as a contextual bandit learner is to demonstrate its simplicity and interpretability. Although the primary motivation of TM is its use as an interpretable function approximator applied to Reinforcement learning, we demonstrate its use in the contextual bandit setup which is perhaps the first step towards the full reinforcement learning problem, which we intend to solve with TM in the future. Indeed there are several methods for variable selection and measuring importance of features, however, TM is a learner that produces propositional expressions for the arms (chosen actions) consisting of context features, i.e. it has the capability of producing interpretable rules for choosing particular actions (not just variable or feature selection/importance). It learns propositional rules to solve contextual bandit problems, which has the added advantage of being interpretable.

2. Both the Tsetlin Machine and the Tree methods are non parametric learners. Tsetlin Machines are rule based like decision trees. However, they are quite different in several aspects such as learning mechanism, learning elements (feature nodes for decision trees and clauses for TM), please see [1] for complete details. TMs are also universal function approximator like neural networks,  and most importantly they are hardware-near with low energy and memory footprint since they are entirely based on Boolean propositional expressions.

$[1]$ Granmo, Ole-Christoffer. "The Tsetlin Machine—A game theoretic bandit driven approach to optimal pattern recognition with propositional logic. arXiv 2018." arXiv preprint arXiv:1804.01508.

$[2]$ Jiao, L., Zhang, X., Granmo, O. C., and Abeyrathna, K. D. (2021). On the convergence of Tsetlin Machines for the XOR operator. arXiv preprint arXiv:2101.02547.

$[3]$ Zhang, X., Jiao, L., Granmo, O. C., and Goodwin, M. (2021). On the Convergence of Tsetlin Machines for the IDENTITY-and NOT Operators. IEEE Transactions on Pattern Analysis and Machine Intelligence.

    

 Thank you for your feedback.
For further details on the TM algorithm, please refer to [3]. TM is universally interpretable on all types of data, including images as shown in [4,5].
We consider a setup, where contexts are i.i.d and the classification oracle is a Tsetlin Machine. We compare Tsetlin Machine as a classification oracle for contextual bandits with other algorithms where we show that one can obtain interpretable results which are simple to understand since it considers proposition logic. The Square CB method uses a regression oracle to predict the score for each action.TM can be used like any other regression oracles with Square CB reduction, but we leave that for future work. 
1. The reason for having clauses with positive and negative polarity is to learn both what the target concept looks like (positive clauses) and what it does not look like (negative clauses). The negative clauses boost the expression power by introducing negation at the concept level, however, only using positive clauses also works well in our experience, although potentially with slightly lower accuracy.
2. Thanks for the sharp observation - you are right, the roles of the odd- and even-numbered clauses should be reversed in the explanation.
3. The ensemble effect increases with increasing T because T decides how many clauses must agree to stop learning a particular pattern. If more clauses need to agree, one gets an averaging effect between the agreeing clauses that increases diversity as well as robustness towards the noise. A T as low as 1 only requires that a single clause captures a particular pattern and then learning stops for that pattern. Then one can capture more patterns with fewer clauses, but the robustness of the representation of each pattern suffers by only being represented by a single clause. 
4. With weighted TM you introduce an integer weight $w_j$ per a clause in Eq. 4, in-between the summation operator and conjunction operator:
    \begin{equation}
    \hat{y} = 0 \le \sum_{j=1,3,\ldots}^{n-1} w_j \bigwedge_{k=1}^{2o} \left[g(a_k^j) \Rightarrow l_k\right] - \sum_{j=2,4,\ldots}^{n} w_j \bigwedge_{k=1}^{2o} \left[g(a_k^j) \Rightarrow l_k\right].
    \end{equation}
    The weight of a clause is incremented when the clause produces a true positive and decremented when it produces a false positive. As a result, the weight captures the precision of its clause. The weighting operates together with $T$, so that a high-precision clauses can make decision more or less on their own, while low-precision clauses need to assemble to approach $T$ (simulating a Naive Bayes like decision). The end result is typically an increase in accuracy, with fewer clauses.
5. When deep neural networks are used as regression oracles in the square CB setup, the underlying mechanism of how the scores are predicted is difficult to obtain. The same goes for which feature played an important role in the prediction of the score. Approaches taken by [1] to generate explanation vectors for feature importance, and the linear proxy model and rule extraction methods [2] are approximate, and only demonstrated for small representations, also adding computation cost when constructing proxy models. Since there is a high preference given to logic as a language of explanation [2] for interpretability, it is desireable to use TM since they provide interpretable results with Boolean logic.
Examples of such assumptions include parametric modelling assumptions in which a parametric relationship between context and rewards are assumed to exist. Therefore one needs to perform feature engineering to transform the data so that it satisfies the parametric modelling framework. Other examples include assumptions where the reward is a linear function of the contexts [6]. These models often perform poorly when estimating probabilities from binary response data.

    [1] G. Montavon, W. Samek, and K.-R. Muller, “Methods for interpreting and understanding deep neural networks,” Digital Signal Processing, vol. 73, 2018.

    [2] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal, “Explaining explanations: An overview of interpretability of machine learning,” in 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), 2018.

    [3] Granmo, Ole-Christoffer. "The Tsetlin Machine—A game theoretic bandit driven approach to optimal pattern recognition with propositional logic. arXiv 2018." arXiv preprint arXiv:1804.01508.

    [4] J. Sharma, R. Yadav, O.-C. Granmo, and L. Jiao. Human Interpretable AI: Enhancing Tsetlin Machine Stochasticity with Drop Clause. arXiv preprint arXiv:2105.14506, 2021.

    [5] O.-C. Granmo, S. Glimsdal, L. Jiao, M. Goodwin, C. W. Omlin, and G. T. Berge. The Convolutional Tsetlin Machine. arXiv preprint arXiv:1905.09688, 2019.
    
    [6] Agrawal, Shipra, and Navin Goyal. "Thompson sampling for contextual bandits with linear payoffs." International conference on machine learning. PMLR, 2013.




 This paper studies the use of Tsetlin Machines for the contextual bandit problem. Tsetlin Machines learn a classifier using propositional logic assuming the given context is also binarized. The key proposed advantage is that such a system yields an interpretable model which is useful for many applications. The paper discusses the background and learning mechanism for Tsetlin Machines and then applies Tsetlin Machines in contextual bandit using epsilon-greedy and Thomson sampling-based exploration approaches. Experimental evidence in several datasets is provided showing that the new methods outperform neural networks using epsilon-greedy and decision trees.  Strength:

1. Interpretability is an important topic in machine learning, and interpretable models are very much sought after in certain domains. The proposed work can be useful, therefore
2. Empirical evidence show the advantage over some other methods.

Weakness:

1. The paper while promising falls short on several details. Firstly, while a discussion of the learning mechanism for Tsetlin machine is presented, very little is offered in why such a learning rule is advantageous. No theoretical results are stated to show that it converges, and no details are given for certain decisions (see questions below for examples).

2. It is unclear why the rules generated by the model are necessary interpretable. If the context was an image and each pixel was somehow transformed into a set of 0-1 features, then a rule that talks about the conjunction of these features would not be interpretable. Such a model will not yield any more insight than a neural network that given an input generates a certain output.

3. Experimental evidence against strong contextual bandit baselines like SquareCB (Foster et al.) which are known to have optimal regret is not presented. My score is tentative and likely to change based on author's response. 

1. Why does Equation 4 consist of the sum of odd-numbered clauses minus the sum of the even-numbered clauses? Can we not simply use a single sum of all clauses and see whether this value is greater than n/2 or less than n/2?

2. On line 83, does voting for anything mean output 1? And Equation 4 seems to suggest that when odd-numbered clauses vote 1, then the sum is greater than 0 and so \hat{y}=1, but the text says that odd-numbered clauses vote for \hat{y}=0. Same on the last line on line 88-89. Do you mean "leading to class prediction \hat{y}=0". 

3. What does this line mean, "Here, T is a user-configurable voting margin yielding an ensemble effect."? How is this leading to an ensemble effect?

4. How is the weight of a clause used in making predictions? What is the analog of Equation 4 for weighted Tsetlin machines? 

5. How does this work compare with methods for interoperability in deep neural networks (e.g., if we use neural networks and a contextual bandit approach like SquareCB)? 

6. On line 32, it says, "make assumptions that limit their usability in real-world settings". Please provide examples of these assumptions. The paper discusses the limitations of binarization of context.

The presentation can be improved. E.g., Algorithms 1 and 2 can be shortened or moved to Appendix and the space freed can be used for more qualitative discussion. The discussion on "Weighted Tsetlin Machine" can use more motivation and an explicit equation showing how the decision is made when clauses have weights. 



 This paper applies Tsetlin Machine into contextual bandit algorithm. 
Because Tsetlin is an inherently interpretable machine learning method, the resulting bandit algorithm is interpretable. 
Empirically the authors tests its performance by nine datasets.  Strength: The paper introduces a new method to the bandit community. 

Weaknesses: (1) The paper lacks the theoretical support.  

(2) Besides of the lack of the theoretical results, the motivation of applying Tsetlin is not clear.  (1) There lacks of further explanation why the interpretability from Tsetlin is important in contextual bandits. 
This is not clear from the introduction.  The second paragraph in the introduction looks messy to relate the interpretability of Tsetlin to contextual bandits. For example, If we concern the features in bandits, there are many methods about variable selection and importance of features. 

(2) About Tselin Machine in Section 3, it looks to be related to tree methods. Are there more information for this relation? I think it's helpful for readers like me to understand this method. 

(3) In Section 4, the authors do not provide the motivation of applying the method to contextual bandits.  It looks like just applying it. This makes the contribution weak. 
 (1) The Section of Tsetlin is a little lengthy but a little redundant.  It's helpful if section 3 could be written concisely. 

(2) For empirically supporting the application of Tsetlin, more empirical studies are needed. Algorithm 2 makes use of boostrap technique in the bandit algorithm, so it's important to show the contributions of boostrap and Tsetlin separately.  The current version is not clear to show them. 

(3) The format of the article is not easy to read. I think it's important for a formal submission. 

 This paper studies the contextual bandit problem using the Tsetlin Machine framework. In detail, the authors propose a method to learn the binary-valued data and a Thompson sampling-based algorithm to balance the exploration for contextual bandits. Experiments are conducted to evaluate the performance of the algorithm compared to several baseline algorithms.  Strengths: studying the contextual bandits with binary contexts under the Tsetlin framework sounds novel to me. Experiments are comprehensive. The paper is carefully written and easy to read.

Weakness: The sampling is conducted by perturbing the collected historical data according to the authors. This raises two major concerns in my view. First, keeping the historical data and randomly changing them in **each** episode sounds inefficient. Second, according to [1] or [2], to guarantee a sublinear regret bound by perturbing historical data, one may need some special warm-up procedures in the algorithm (see lines 5-9, Alg. 1 in [1], lines 4-8 Alg. 1 in [2]). I wonder if the same procedure should be applied here to guarantee a good result.

[1] Kveton, Branislav, et al. "Perturbed-history exploration in stochastic linear bandits." arXiv preprint arXiv:1903.09132 (2019).
[2] Jia, Yiling, et al. "Learning neural contextual bandits through perturbed rewards." International Conference on Learning Representations. 2021. What's the space complexity of the algorithm? How will the number of bits during the binarization and the coding methods will affect the performance of the algorithm? NA

Meta-review: Thank you for submitting your paper to NeurIPS! The authors introduce a new contextual bandit algorithm using propositional logic (Tsetlin Machines), offering a more interpretable learning approach. The method interfaces with existing bandit algorithms like Thompson Sampling and \epsilon-greedy, and achieves favorable empirical results in an extensive suite of experiments. While the paper does not provide any technical regret guarantees, the review team found the empirical results intriguing and could lead to future work. I am pleased to recommend acceptance.

Decision: Accept