abstract: The study of opinions $-$ e.g., their formation and change, and their effects on our society $-$ by means of theoretical and numerical models has been one of the main goals of sociophysics until now, but it is one of the defining topics addressed by social psychology and complexity science. Despite the flourishing of different models and theories, several key questions still remain unanswered. The aim of this paper is to provide a cognitively grounded computational model of opinions in which they are described as mental representations and defined in terms of distinctive mental features. We also define how these representations change dynamically through different processes, describing the interplay between mental and social dynamics of opinions. We present two versions of the model, one with discrete opinions (voter model-like), and one with continuous ones (Deffuant-like). By means of numerical simulations, we compare the behaviour of our cognitive model with the classical sociophysical models, and we identify interesting differences in the dynamics of consensus for each of the models considered.
address: $^1$ Laboratory of Agent Based Social Simulation, Institute of Cognitive Science and Technology, National Research Council (CNR), Via Palestro 32, 00185 Rome, Italy.; $^*$ Correspondence and requests for materials should be addressed to D.V. (email: email@example.com).
author: Francesca Giardini$^1$, Daniele Vilone$^{1,*}$, Rosaria Conte$^1$
bibliography: GVC15_arXiv_revised.bib
title: Consensus Emerging from the Bottom-up: the Role of Cognitive Variables in Opinion Dynamics

KEYWORDS: Opinion dynamics, cognitive modeling, social influence, agent-based modeling, sociophysics.

# Introduction

Opinions represent a large part of human mental representations, and a large part of our everyday social interactions consist in exchanging, evaluating, revising and comparing opinions with our family, friends, acquaintances, or even strangers. Understanding opinions, describing how they are generated and revised, and how far opinions travel across social space both as a consequence of social influence and as one of the main means through which social influence unfolds, is crucial for grasping a deeper understanding of human social cognition and behaviours.

Since the beginning of the last century, sociologists and psychologists have been interested in understanding opinions, focusing in particular on the multiplicity of dimensions that characterise this phenomenon . Other disciplines have also been interested in the topic, like sociophysics and complexity science . In the last decades, the study of social phenomena as opinion formation and dynamics has become of great interest in physics. Due to similarities between spreading and ordering phenomena, opinion dynamics has been studied from a mathematical and numerical point of view by means of the tools of statistical and computational physics . In particular, in the physics community, opinions have been so far considered dynamic elements that can be approximated as spin systems or by similar statistical-mechanical methods . Moreover, the possibilities offered by Big Data Science to collect and analyze huge amounts of digital traces humans leave on the web and other media, has made opinion change and consensus achievement exceeds disciplinary boundaries and become one of our century's grand scientific challenges .

Nonetheless, a theory explaining micro-foundations of opinion dynamics is still missing. In cognitive terms, opinions are highly dynamical mental representations resulting from the interplay between different mental objects within individuals' minds. Another distinguishing feature is their being easily affected by opinions of other individuals in the same network, as we will show in our model. Without explaining how opinions are formed and manipulated within the individuals' minds, it is very difficult to account for the way in which they change as an effect of social influence. Even more difficult is to predict consensus or to identify mechanisms leading to polarization or to isolation and/or integration of minorities. This means that when explaining the emergence of macro-social phenomena we need to know what happens at the micro-level, *i.e.*, what drives human actions and decisions in order to understand how individuals' representations and behaviours can give rise to socially complex phenomena and how those affect agents' actions. Defining an opinion in terms of its mental ingredients and specifying how human cognition promotes opinion change or resistance provides not only a more realistic description of the phenomenon of interest, but it represents an attempt to model the bottom-up emergence of opinions' persistence, the effects of contrasting forces on them and the generation of alternative paths of diffusion.

We aim at developing a cognitively grounded model of opinion dynamics that will allow us to answer the following questions: can we identify the distinctive features of opinions, and model them as interacting representations that get influenced by others' mental states? How can heterogeneous agents, endowed with different representations of the external world, come to share a given viewpoint and what consequences this sharing has on individuals' beliefs and their related behaviours?

Aim of this paper is to put forward a new model of opinion dynamics in which opinions are defined by three main features that interact and support opinion revision and change. We test our model through simulation experiments with two different network topologies and we try to identify how changes at the micro-level of agents' cognition can give rise to changes in opinions at the collective level. In order to check for our models' robustness and reliability, we will also compare our results with existing models, in particular the Voter  model and the work of Deffuant .

## Opinions in the Social Sciences

The study of opinions' formation and spreading originated within the field of social science, in which several important contributions were developed. A comprehensive review of the social and psychological literature on the topic is beyond the scope of this work, but in this section we will tackle the background and we will review some of the main theories that social scientists put forward about opinions and opinion change with the aim of situating our inter-disciplinary model in the wider context of the social sciences.

In general, opinions are treated as synonyms for different mental objects, as beliefs , or more frequently, attitudes . In general, opinions and attitudes are used as interchangeable terms referring to a mental object liable to social influence and persuasion . The specificity of attitudes and the fact that they cannot be considered opinions is made clear in Allport's work . In this account, attitudes define and shape people's perceptions, whereas opinions result from a more articulated process in which beliefs, goals, intentions and knowledge play a role. An attitude can be regarded as a necessary element in opinion formation, but it is not sufficient. Allport recognizes the difference between attitudes and opinions, but he nonetheless considers the measurement of opinions as one way of identifying the strength and values of personal attitudes. An alternative view contrastss the affective content of attitudes with the more cognitive quality of opinions that involve some kinds of conscious judgments . In general, it is possible to identify two main trends in the relevant literature.

On one hand, researchers focus on attitudes, considered as implicit evaluations that allow to access individuals' positive or negative views about given matters, and that are supposed to be stable across time and relevant enough to predict individuals' behaviors and actions. On the other hand, explicit theories of opinions are more centered on conscious reasoning and judgment, through which individuals are supposed to form and express measurable opinions.

Another general feature of the social and psychological approach is the preminence given to measuring, rather than conceptualizing opinions. As a result, many studies (for a review, see ) tried to develop reliable and fine-tuned ways to measure people's approaches to general questions, partially abandoning the issue of defining what an opinion is, and focusing on how it should be measured. Quantifying opinions and measuring their diffusion in a given population is important in order to understand, monitor and predict social and political events, but it is not yet an answer to the question: What is an opinion? Answering this question is essential to define good measures, to identify the right tools and also to get insights from the results. If the measurement does not take into account the main features of opinions, then we will never know what we are exactly measuring and how good our predictions can be, based on that measure.

In political science, Crespi considers individual opinions as "judgmental outcomes of an individual's transactions with the surrounding world" (p.19), emphasizing the interplay between what he calls an attitudinal system and the external world characterized by the presence of other agents and different subjective perceptions. Opinions are the outcomes of a judging process but this does not mean that they are necessarily rational or reasoned, although Crespi recognizes that they need to be consistent with the individual's beliefs, values and affective states. It is worth noticing that many contributions are specifically oriented to understand "public opinion" , as the collective result of integration of opinions and attitudes coming from different sources and exposed to different kinds of influencing.

As other authors already pointed out , many models of opinion and social influence do not provide careful definitions of what an opinion is and how it is affected by social influence. This happens to be true also for theories of persuasion, like the social impact theory , a theory of how social processes operate at the level of the individual at a given point in time. Part of this theory has been developed using computational modeling by Nowak, Szamrej and Latané . In their model, individuals change their attitudes as a consequence of other individuals' influence. In parallel with the idea that social influence is proportional to a multiplicative function of the strength, immediacy, and number of sources in a social force field suggest that each attitude within a cognitive structure is jointly determined by the strength, immediacy, and number of linked attitudes as individuals seek harmony, balance, or consistency among them. Therefore, a single opinion does not exist in a "vacuum" and its change depends on the interplay of several factors. Their results show that group dynamics are dependent upon initial spatial configurations, as well as from the different parameters, hence opinion change is completely dependent upon interactions happening at the group level.

## Tackling the sociophysical background

The first sociophysical approach to opinion dynamics was the voter model (VM). It was originally conceived   as a model of competition of species, but it was soon adapted to model an electoral competition between two candidates . The VM dynamics is extremely simple: each agent is endowed with a binary variable which can assume one of two distinct values. In each elementary interaction an agent is randomly selected and assumes the opinion of one of its neghbours (again randomly selected). Initial conditions are also generally set at random. This model has been thoroughly considered in sociophysics because it can be solved exactly in every dimension  and thanks to its simplicity and flexibility has been used as a starting point in the quantitative treatment of social phenomena.

In general, the VM dynamics can freeze only with consensus, that is, when all the agents come to share the same opinion. In fact, only in one and two regular dimensions the system is actually driven towards consensus, following a power-law decay in 1D and a logarithmic one in 2D. In higher dimensions and on many complex networks the system is not able to reach consensus and remains in a disordered metastable state in the thermodynamical limit  (finite size systems eventually reach consensus thanks to statistical fluctuations, after a freezing time which diverges rapidly with size).

Interestingly, on small-world networks, the level of cooperation of the metastable state is proportional to the density of long-range connections among distant parts of the system .

However, its simplicity is also one of the main limitations of the model. Attempts to overcome its limits included the application of the original VM in more realistic configurations, meaning that the VM was tested in different kinds of complex topologies . Another interesting direction consisted in adding agents with special features with the aim of explaining fundamentalisms  or the effects of mass media .

The presence of fundamentalists or "zealots" who never change opinion, in general does not enhance consensus: actually, a zealot forces the whole system towards its opinion in one and two regular dimensions, where consensus is anyway achieved. On the contrary, in different topologies we observe the presence of a small region of localized consensus only in the neighborhood of the fundamentalist .

In the VM interactions are based on imitation, a drastic simplification of human interactions. In order to make interaction more realistic,   proposed a majority rule (MR): at each elementary time step a group of individuals is picked up at random, then all the members adopt the opinion of the majority of the group itself. This rule was proposed to describe public debates and interactions among individuals belonging to different groups. Also with the majority rule, the system can end up to consensus or to a metastable disordered state, depending on the details of the topology and the initial conditions.

In socio-physics, a great deal of work focused on the operational definition of opinion itself. For example, it can be allowed to assume more than two values  or defined as a continuous variable . These models present a richer dynamics and the number of possible active and frozen states accessible to the system is much larger, so that the phenomena observed are in principle more realistic. Among these models, the Deffuant Model (DM)  deserves attention. Here, an opinion is defined as a real number ranging from 0 to 1: therefore, it can be somehow considered a generalization of the VM. A fundamental feature of DM is the threshold in opinion distance for which only agents whose opinions are less distant than a given threshold can interact and then get even closer opinions. This mechanism resembles the Axelrod's model of cultural spreading , where agents are able to interact proportionally to their actual cultural similarity. This leads to a final state with a set of different clusters, each one consisting of agents sharing the same opinion. The number of final clusters depends on the initial conditions and on the model's parameters, especially the distance threshold . Subsequently, the DM has been furtherly refined, for example by defining another variable characterizing the "affinity" among agents , so that opinions and affinities coevolve: if there is a sufficient affinity between agents, their opinions become more similar, and viceversa. In this version of the model, consensus is reached more easily than in the original DM.

Although different in several respects, the models mentioned above share the same feature: opinions are given from the beginning and evolve exclusively in interaction with other agents and on the basis of the environment, without any internal characterization. No hint of what happens within individuals' minds, how opinions may emerge and evolve internally, is given: in practice, in sociophysics models we have interacting opinions rather than interacting individuals.

Aim of this work is to overcome the limitations of the social sciences, in which no operational models of opinions are provided, and the limitations of socio-physics just discussed. Moving from a definition of opinions as cognitive representations, we will use socio-physics tools in order to put forward an inter-disciplinary model of opinions which takes into account the internal dynamics of individuals' minds and can be used to analyze opinion spreading and consensus formation.

## A tripartite model of opinion: truth-value, confidence and sharedness

In this work we want to propose an integrated account in which we identify not only the specific cognitive features that characterise opinions, but also the way in which each single feature gets modified by interacting with others' opinions. A cognitive model of opinions and their dynamics requires to provide a definition of opinions as mental representations and to present specific features that make their revision and update easy and enduring. Moreover, grounding opinions in the mind allows us to take into account not only direct processes of revision due to the interactions with others (social influence), but also revisions based upon changes in one's own mental representations supporting that opinion.

On the basis of the previous considerations, opinions are mental representations characterised by the following features:

-   subjective truth-value

-   confidence

-   perceived sharedness.

We assume that objective truth value cannot be verified (or it is not relevant), so we refer to a "subjective truth-value" that expresses whether and how much someone believes an opinion to be true. Opinions are epistemic representations that refer to matters or events that cannot be defined as either true or false because it is not possible to verify alternative states of the world , as for example the consequences of an event that did not happen, or the outcomes of an election which has not been held yet. In many cases opinions are emotionally loaded, and individuals have positive or negative attitudes that reinforce their opinions, as for instance about abortion . In that case there may be not any ascertainable truth about it, but people often have strong opinions on the basis of their beliefs, culture or personal experience. In order to convert a subjective truth-value into a practical quantity, we will express it as a variable indicating how true an opinion is considered by a given agent.

The second defining feature of our model is the degree of confidence. This is a subjective measure of the strength of an opinion and it expresses the exent to which one's opinion is resistant to change, like a sort of "opinion inertia". When an individual is highly confident, he has a number of reasons to believe that his/her opinion is right, and the higher the confidence, the more willing that individual is to defend his/her opinion against others'. For the sake of simplicity, confidence is currently expressed as an arbitrarily assigned value. It is worth stressing the difference between subjective truth-value and degree of confidence: an individual can be somehow uncertain about his/her opinion, but struggle fiercely for it ("right or wrong, it's my country"); or, viceversa, he can be absolutely certain about a given issue without defending it publicly (for fear or lack of interest).

The third feature of our model is perceived sharedness, *i.e.*, the extent to which a given opinion is perceived to be shared. This subjective measure allows us to distinguish between the actual number of individuals having the same opinion and the personal experience that an agent can have. Believing that there are a number of other individuals sharing my opinion does not mean that this is actually the case, actually I can be part of a minority and not be aware of the opinion of the majority. Perceived sharedness may heavily affect degree of confidence, making people feel more confident because they might feel supported by the majority. On the other hand, this factor can also lead an individual to revise his/her opinion because confidence in it is low and it is also perceived as a minority opinion.

Modeling opinions as results of the interplay among the above defined internal variables allows us to take into account their distinctive nature of mental objects. Knowing how opinions change within individuals' minds is essential to understand whether this change can be robust, how far it can travel into a given social group and how far it can spread. Unlike other models that treat opinion change as an effect of plain social influence , we are interested in modeling what happens inside individuals' minds. Opinions do not change suddenly, but they result from interacting internal dynamics and external influences, as our model is aimed to show.

# Discrete Opinion Model

## Description

We set $N$ individuals which are characterised, at every time step $t$, by their opinions $O_i(t)$ ($i=1,\dots,N$). In the discrete version of the model, opinions can assume one of two possible values, 0 or 1. Agents are characterised by three internal variables referring to their opinions: subjective truth-value $x_1^i(t)$, degree of confidence $x_2^i(t)$, and perceived sharedness $x_3^i(t)$. These variables are defined as continuous quantities, ranging from 0 to 1.

Agents are located on the nodes of a graph, with the relationships among them given by the links. At the beginning of the simulation, agents are initialized as having one of two opinions which are randomly assigned with equal probability (with some exceptions which will be pointed out when needed). Internal variables are assigned at random as well, following a uniform distribution. For each elementary time step, two agents are randomly chosen and they are attributed one of two roles: a listener, say $i$, and a speaker, $j$. Speakers are selected among listener's neighbours. In each interaction, evolution applies only to listeners, which, in our asymmetric model, are those who can revise their opinions. It is worth noticing that in our model, even though two agents share the same opinion, this can result from completely different internal variables. In particular, opinions and internal variables coevolve. People sharing the same opinion reinforce their beliefs when they meet, conversely interacting with an agent of opposite opinion may drive individuals to change it or at least to be less sure of their initial opinion.

In our model, the subjective truth-value and the degree of confidence of a listener increase (or remain unchanged) when interacting with a speaker sharing the same opinion. In formal terms, we implement the dynamics of these internal variables when $O_i(t)=O_j(t)$ as follows

1.  If $x_1^i(t)<x_1^j(t)$ and $x_2^i(t)<x_2^j(t) \Longrightarrow x_1^i(t+\Delta t)=x_1^j(t)$;

2.  If $x_1^i(t)\geq x_1^j(t) \Longrightarrow x_1^i(t+\Delta t)=x_1^i(t)$;

3.  If $x_2^i(t)\geq x_2^j(t) \Longrightarrow x_2^i(t+\Delta t)=x_2^i(t)$;

4.  If $x_2^i(t)<x_2^j(t) \Longrightarrow x_2^i(t+\Delta t)=\frac{x_2^i(t)+2x_2^j(t)}{3}$.

In practice, when a listener is paired with a speaker sharing the same opinion, its subjective truth-value cannot decrease. In particular, we assume that if the speaker is more confident than the listener, the latter is reinforced in his/her opinion; vice versa, a speaker with a low degree of confidence can marginally influence a listener. We implemented this rule in the simplest way: the listener will assume the speaker's subjective truth-value ($x_1^j$) if and only if both confidence and subjective truth-value are larger in the speaker, otherwise $x_1^i$ does not change. Rules 3 and 4 define how confidence changes as an effect of interactions. If a listener meets a more confident speaker with the same opinion, this will increase the listener's confidence, otherwise it will not be affected. Unlike subjective truth-value, the listener does not assume the speaker's confidence, but there is an adjustment between the two values.

Let us now define the rules for the dynamics of the opinion and the internal variables when a speaker $j$ and a listener $i$ of different opinions meet. In this case, opinion change depends on confidence and perceived sharedness. In general, we assume that people can be better persuaded by more confident speakers, in line with evidence in social psychology showing that consistent and confident minorities can influence majorities and make them revise their judgements . We also consider that there is a positive correlation between the speaker's subjective truth value and the probability of convincing the listener. Finally, humans are sensitive to social pressure, then believing that one's own opinion is shared by the majority makes opinion change less likely . Therefore, we define a simplified rule such that the probability $P_{ij}$ that $i$ assumes the $j$'s opinion is zero if speaker's confidence is smaller than the listener's, otherwise it is directly proportional to the speaker's subjective truth-value and to the quantity 1 minus the perceived sharedness. This points to the fact that an agent is reluctant to change its opinion if thinking to share it with the majority. Summarizing, it is

$$P_{ij} =
\left\{
\begin{array}{ccc}
x_1^j(t)[1-x_3^i(t)] & \mbox{if} & x_2^j > x_2^i \\
 & \\
0 & \mbox{if} & x_2^j\leq x_2^i \ ;
\end{array}
\right.
\label{pij}$$

Afterwards, if the listener shifts towards the speaker's opinion, we assume that the former also acquires the subjective truth-value of the latter, but with a confidence level equaling half of the speaker's confidence.

$$\begin{array}{l}
x_1^i(t+\Delta t)=x_1^j(t) \\
 \\
x_2^i(t+\Delta t)=x_2^j(t)/2 \ .
\end{array}
\label{int1}$$

On the contrary, if $i$ does not change opinion, the listener's subjective truth-value remains unchanged, but the interaction affects in any case the confidence. More precisely, we assume that interacting with someone with a different opinion reduces the listener's degree of confidence. Therefore, we set the new listener's confidence as the average between its own confidence before the interaction and the speaker's confidence level in case the latter was smaller, to half its original value if initially it was $x_2^i<x_2^j$.

$$x_1^i(t+\Delta t)=x_1^i(t)
\label{int2_a}$$

$$x_2^i(t+\Delta t)=
\left\{
  \begin{array}{cl}
\frac{x_2^i(t)+x_2^j(t)}{2} & \ \ \mbox{if}\ \ \ \ \ x_2^i(t)>x_2^j(t)  \\
 & \\
\frac{x_2^i(t)}{2} & \ \ \mbox{if}\ \ \ \ \ x_2^i(t)\leq x_2^j(t)  \ .
\end{array}
\right.
\label{int2}$$

Finally, we implemented perceived sharedness $x_3^i(t)$ as the total number of past encounters with other agents sharing the same opinion an agent $i$ has up to time $t$, independently of the outcomes of each interaction. More precisely, each agent records all past encounters as a listener, determining the frequency of both opinions: the perceived sharedness at time $t$ is the frequency of the opinion shared by the agent itself at the same instant $t$.

Time is measured in Monte Carlo steps. This means that a single time unit is made up by $N$ single interactions, *i.e.* $\Delta t\equiv1/N$.

Certainly, this model must be seen as a first step, and further studies and in-depth analyses will be useful for a more precise definition of its parameters.

## Topology

We tested the behaviour of our model on two different topologies: a total connected graph (mean field) and a one-dimensional ring. We chose them in accordance with the relevant literature , in which these topologies are usually considered as starting points. Moreover, given the complexity of the relationships among agents' internal variables, even simple topologies can be useful in understanding the basical properties of the model. More complex topologies will be used in future works.

## Results

For each version of the model, we initially considered a system of $N=100$ agents and tested its behaviour starting from three different initial distributions of agents holding two different opinions: $\langle O_0\rangle=0.5,\ 0.75$ and $0.9$. The last two distributions represent an attempt to model a situation in which there is a conflict between a majoritarian and a minoritarian opinion. We stress the fact that every measure represents an average over 2000 independent realizations. More details are given in the following subsections.

### Mean field

In every simulation the system ends up to a substantial consensus. At the end of each realization, all the agents share the same opinion, even if there are interesting differences between individuals' mental states.

In Figure  we show how average opinion behaves over time. It is worth noticing that an initial asymmetry, even if small, allows the majoritarian opinion to invade the whole system and to become dominant. That means that the opinion which at $t=0$ is majoritarian, even slightly, always ends up being the only survived one. This result contrasts with the classical mean field voter model, in which the initial average opinion is always conserved .

On the other hand, in Figure  we show the behaviour over time of the averaged internal variables with $\langle O_0\rangle=0.5$. Among these, perceived sharedness shows a noticeable behaviour: it takes longer to reach its final level (which equals 1 since all the agents share the same opinion eventually). In other words, the system reaches consensus even before agents realize they all agree, showing an interesting dynamics between the micro- and the macro-level. This result is remarkable especially if we think of real-world situations, in which there is usually a gap between what individuals know locally and the global and emergent behaviour of the system in which they are embedded. This delay is due to the early encounters at the initial stages of the dynamics, kept in memory by the agents, which drive the individuals to mantain a distorted idea of the others' views for a longer time. It is worth to notice that the persistence of the initial opinion and impressions in human mind is a well-known phenomenon, already discussed in the psychological literature .

In order to be sure that the observed results do not depend on the size of the system, we also repeated the same simulations with $N=250$: we did not observe any significant differences increasing the number of agents in the population.

It must be noticed that even if this dynamics leads the system to consensus, in half of the realizations there is one agent who maintains its minoritarian opinion. This resistance to social influence happens when the agent with the largest initial degree of confidence has the opposite opinion of the majority: from Equation () follows that such agent can never change opinion. Such a result could help explaining extremisms in real-world scenarios, where no matter the kind of social influence and persuasion someone is exposed to, there are always individuals who stick to their opinions.

### One-dimensional ring

We tested our model also on a one-dimensional ring with the same number of agents ($N=100$).

Figures  and  show how average opinion (for three different initial conditions) and internal variables (for $\langle O_0\rangle=0.5$) change over time, respectively. Concerning the average opinion, qualitatively these results are similar to the mean field case, even though here the invasion of the initially majoritarian opinion is much slower. This behaviour is confirmed in Figure , where we compare the time behaviour of active bonds (*i.e.* the links between two agents with opposite opinion) in our model and in the one-dimensional voter model , in which, at each time step, a randomly selected agent simply imitates the opinion of one of its neighbours. This makes the system reach consensus through a power-law decay with exponent $1/2$ (and a final quick exponential convergence time depending on the square of the system size), whereas a different behaviour is observed in our model. After an initial power-law decay with exponent $\beta$ close to $2/3$ (a numerical fit reports $\beta=0.65\pm0.01$), the active bond density decreases much more slowly (maybe tending to a steady state), as shown in the inset of the same figure.

# Continuous Opinion Model

## Description

We also present another version of the model in which opinions are continuous, so that an opinion is a real variable ranging between 0 and 1 (initially randomly assigned to each agent following a uniform probability distribution). The model dynamics is the same we described above, but we extended its scope by adding a new rule for opinion revision. Here, when the listener accepts the speaker's opinion, opinion changes in accordance with Deffuant's rule :

$$O_i(t+1)=O_i(t)+\mu[O_j(t)-O_i(t)] \ ;
\label{contin}$$

with $\mu=0.5$. In order to determine the perceived sharedness, we consider opinions up to 0.5 as of "negative" kind and from 0.5 as "positive", so that $x_3^i(t)$ is the number of past encounters with other agents sharing the same kind of opinion an agent $i$ has up to time $t$, starting from $t=0$. The internal variables are assigned at random, following a uniform distribution, also in this version of the model.

## Results

### Mean field

In the continuous version of the model, when the graph is totally connected, the system reaches final consensus, even though conserving the average initial opinion, as we can see in Figure . On the other hand, internal variables behave as in the discrete case, as shown in Figure . This means that when we apply a mean field topology, the model shows the same behaviour, regardless of the way in which opinions are implemented (either as discrete or as continuous variable): the system achieves final consensus with high levels of subjective truth-value and degree of confidence, reached even before the agents realize they have already ended up sharing the same opinion.

# Discussion

Opinions are a complex issue for a variety of reasons and they represent a very interesting case of the micro-macro link: opinions are mental objects that get modified by social processes and then re-enter the mental space. More specifically, internal representations of individuals, such as beliefs, goals, and intentions, give rise to the complex dynamics at the macro-level, which feed backs into the lower level. However, pivotal to understanding the dynamics of opinions is the definition of specific features, which characterise how opinions are created and modified within individual minds.

In this work, we have outlined the micro-level of opinions and started to explore their dynamics. We identified three main constitutive features of opinions within individuals' minds. The first internal variable is the subjective truth-value, which describes how much the individual believes her/his opinion to be true. Saying that opinions have a subjective truth-value does not mean that individuals do not believe in them or that they cannot be strongly opinionated. This feature is crucial because it accounts for the fact that opinions get more frequently influenced by interactions with others through social influence, but they can also be easily revised according to changes in one's own mental representations, without any external influence. The second feature is what we called degree of confidence, which defines to what extent an individual trusts his/her opinion and how much the opinion is resistant to change. The last variable we introduced is perceived sharedness, *i.e.*, the extent to which one believes others share the same opinion, thus providing support for it. Perceived sharedness does not necessarily overlap with the actual sharedness, and there can be a gap between how much someone believes an opinion to be shared and the actual number of individuals sharing it.

Our results show that the interaction of these variables at the micro-level generates interesting macro-dynamics. Compared with the voter model, we observe interesting differences. The voter model dynamics has two main features: the magnetization is conserved, but in a mean field topology and one dimension, consensus is reached for every finite system size . This means that, even though in each realization all agents end up sharing the same opinion (0 or 1), the average over the ensemble always conserves the initial average opinion. In our discrete model, instead, as shown in Figure , this is true only for totally random initial configurations (*i.e.*, $\langle O_0\rangle=0.5$). On the contrary, if simulations start with an even small asymmetry between the two opinions, the most spread one outcompetes the other.

Another difference between the voter dynamics and our cognitive model consists in the amount of time needed to reach the final totally ordered state, which strongly depends on the system topology. On a totally connected graph our model is faster than VM. Actually, even though the convergence to the final configuration is in both cases exponential, the distribution of consensus times $\tau$, *i.e.* the time needed to reach the consensus, is rather different, as shown in Figure (). Both distributions have indeed the same mode and an exponential tail, but due to its longer tail, the average $\tau$ for the voter model is almost two times larger. Moreover, the latter has a much larger standard deviation ($\sigma_{\tau}\simeq44$ for the voter model, and ($\sigma_{\tau}\simeq5$ for the cognitive one), meaning that also consensus times very far away from the average are likely to be observed.

Conversely, in one dimension we observe that our cognitive model is slower than the voter one. Actually, as it is easy to understand from Figure , the active bond density starts decaying similarly to the voter model (even though with a different power-law exponent), but after a certain time $t^*$ it reaches a sort of plateau whose level decreases slightly on the system size. In short, for our model the time behaviour of active links can be written as follows:

$$n_a(t) \propto
\left\{
\begin{array}{lcc}
t^{-\beta} & \mbox{if} & t\lesssim t^* \\
 & \\
g(t) & \mbox{if} & \ \ t\gtrsim t^* \ ,
\end{array}
\right.
\label{scaling2}$$

where $g(t)$ is a decreasing function of time. From Figure , it is possible to infer that the decay exponent is $\beta\simeq0.65$. On the other hand, the behaviour of $g(t)$ and the value of $t^*$, which *a priori* could also depend on $N$, are very hard to be assessed precisely. Their determination would require the numerical study of much larger systems up to very large times, averaged over more than thousands of realizations, but such effort would go beyond the scope of this work. Anyway, while an exact form of the Equation () may be obtained only through further investigations, the main trend (that is, a slow convergence to the consensus with cognitive dynamics of opinions) is already clear.

In mean field topology our model shows that final consensus is achieved faster, and this is true even when taking into consideration continuous opinions. Actually, comparing Figure  with the results for similar systems reported in , it can be easily noticed that the latter needs much more time to reach the final ordered state, if compared with our results.

Our work represents a first attempt to merge two related but usually distant approaches. Cognitive modeling allowed us to define the basic elements of an opinion as a mental representation characterised by specific features, whereas we used the tools of socio-physics to model the dynamics of opinion spreading. Further investigation is needed in order to better understand opinion change and to set more properly the parameters at stake, but also to predict whether these changes will be stable and under what conditions. In particular, we need to investigate more exstensively the interplay among different dynamics. Opinions change in the mind possibly under social influence, and the social cognitive dynamics of opinions affects the way they spread and the configuration of the space of opinions.

On the other hand, the initial topology may bear consequences on opinion change and on mental features that characterise existing opinions. Furthermore, the interplay between opinions and other mental objects, like other types of beliefs, ought to be addressed, and the external validation of the cognitive model against real-world data should be also achieved. Finally, a well-determined comparison between cognitive and classical socio-physical opinion models, here outlined, would be useful to understand more correctly merits and limits of both approaches.

# Disclosure/Conflict-of-Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

# Author Contributions

Francesca Giardini and Rosaria Conte developed the model, Daniele Vilone performed numerical simulations and analyzed data, and all the authors wrote the paper.

# Acknowledgments

This material is based upon work supported by CLARA (CLoud plAtform and smart underground imaging for natural Risk Assessment), funded by the Italian Ministry of Education and Research (PON 2007-2013: Smart Cities and Communities and Social Innovation; Asse e Obiettivo: Asse II - Azione Integrata per la Società dell'Informazione; Ambito: Sicurezza del territorio).

# References


abstract: Predictions and predictive knowledge have seen recent success in improving not only robot control but also other applications ranging from industrial process control to rehabilitation. A property that makes these predictive approaches well suited for robotics is that they can be learned online and incrementally through interaction with the environment. However, a remaining challenge for many prediction-learning approaches is an appropriate choice of prediction-learning parameters, especially parameters that control the magnitude of a learning machine's updates to its predictions (the *learning rates* or *step sizes*). Typically, these parameters are chosen based on an extensive parameter search—an approach that neither scales well nor is well suited for tasks that require changing step sizes due to nonstationarity. To begin to address this challenge, we examine the use of online step-size adaptation using the Modular Prosthetic Limb: a sensor-rich robotic arm intended for use by persons with amputations. Our method of choice, Temporal-Difference Incremental Delta-Bar-Delta (TIDBD), learns and adapts step sizes on a feature level; importantly, TIDBD allows step-size tuning and representation learning to occur at the same time. As a first contribution, we show that TIDBD is a practical alternative for classic Temporal-Difference (TD) learning via an extensive parameter search. Both approaches perform comparably in terms of predicting future aspects of a robotic data stream, but TD only achieves comparable performance with a carefully hand-tuned learning rate, while TIDBD uses a robust meta-parameter and tunes its own learning rates. Secondly, our results show that for this particular application TIDBD allows the system to automatically detect patterns characteristic of sensor failures common to a number of robotic applications. As a third contribution, we investigate the sensitivity of classic TD and TIDBD with respect to the initial step-size values on our robotic data set, reaffirming the robustness of TIDBD as shown in previous papers. Together, these results promise to improve the ability of robotic devices to learn from interactions with their environments in a robust way, providing key capabilities for autonomous agents and robots.
author: Johannes Günther $^{1,2,*}$, Nadia M. Ady $^{1}$, Alex Kearney $^{1}$,
Michael R. Dawson $^{2, 3}$ and Patrick M. Pilarski $^{1,2, 3}$
title: Examining the Use of Temporal-Difference Incremental Delta-Bar-Delta for Real-World Predictive Knowledge Architectures

# Predictive Knowledge For Robotics

Autonomous agents in the real world face many challenges when interacting with and learning from the environment around them, especially if they are deployed for extended periods of time. As the real world is non-stationary and complex, many of the challenges facing a deployed agent cannot be completely foreseen by its designers in advance. An agent should therefore construct its understanding of the environment using an approach that is continuous and independent, so it is empowered to adapt to its environment without human assistance.

Predictive knowledge is such an approach, and allows autonomous agents to incrementally construct knowledge of the environment purely through interaction . In a predictive knowledge architecture, the environment is modelled as a set of forecasts about how signals of interest will behave. As an agent's actions have an effect on the environment, these forecasts about what will happen next are made with consideration to a policy of agent behaviour (*nexting*, as described by ). In this way, these predictions can capture forward-looking aspects of the environment such as, "If I continue moving my arm to the right, how much load do I expect my elbow servo to experience?" For a concrete example of predictions being used to support robot control, we consider the idea of Pavlovian control, as defined by , wherein learned predictions about what will happen next are mapped in pre-defined or fixed ways to changes in a system's control behaviours. As a principal case study, showed how a sensor-limited robot could use a learned prediction about an impending collision to take evasive action and reduce strain on its motors *before* a collision actually occurred. Without using predictions to alter actions, a collision would need to occur before the robot would be able to take action in response to it.

Detailed demonstrations of the potential of predictive knowledge architectures in real-world domains have been offered in industrial laser welding , robot navigation , animal models of partial paralysis , and artificial limbs . Recently, work has focused on using predictive knowledge to construct representations of state that capture aspects of the environment that cannot be described by current observations alone , and on accelerating the learning of predictive knowledge through the use of successor representations .

From a computational perspective, there is strong evidence that a predictive knowledge architecture is feasible at scale. Many predictions can be simultaneously made and learned online, incrementally , as a system is interacting with the environment, using methods such as temporal-difference (TD) learning and other standard learning algorithms from the field of reinforcement learning. Predictive knowledge architectures have been demonstrated to scale well and to allow real-time learning .

Although research to date has comprehensively established how an agent can utilize prediction learning in a broad range of environments, it is important to note that in all these previous examples, the algorithm for learning is fixed before deployment and does not itself change during learning. Specifically, the step sizes (learning rates) used by the learning algorithms in existing studies are hand-selected in advance by the experimenters through large parameter sweeps or empirical tuning. In addition to the impracticality of hand-selecting learning algorithm parameters, using a predefined and fixed step size for the lifetime of an agent might in fact significantly limit the learning capability of the agent.

It is natural to expect that the learning rate of a long-lived agent should change over time. The process of destabilizing memories and making them prone to change is observed in mammals and is analogous to a temporary increase in learning rates in an autonomous agent. Such a mechanism would be especially useful in a lifelong learning setting , where an agent is confronted with a changing environment and parameters cannot be optimized in advance. Following this idea, recent research has investigated approaches capable of online step-size adaptation , wherein a learning agent is able to self-tune the size of the learning steps it takes in response to the errors observed during its own learning process. However, the aforementioned step-size adaptation methods still use a single step size for all inputs and therefore treat all inputs to a learning agent equally. Not surprisingly, the reliability and variability of different inputs can play a large role in an agent's ability to learn about future outcomes—inputs are not all created equal in terms of their utility for a learning agent. The use of a single scalar step size therefore limits an agent's ability to adapt to and learn more about interesting inputs and to learn less about uninteresting or noisy inputs.

There are several learning rate adaptation methods that modify each individual step-size. AdaGrad, RMSProp, and AMSGrad are methods for deep learning. Temporal-difference learning in and of itself has no relation to neural networks. We can view a neural network as a function approximator for a TD learning method; however, this is one of many function approximators that could be chosen. For instance, the methods used in use a binary recoding of features, similar to the methods in this paper. Appealing to intuition for a moment, when we consider the strengths of RMSProp, ADAM, and others, it's predominantly useful in helping with difficult optimization surfaces for stochastic gradient descent. In part, what makes RMSProp and ADAM so successful is their ability to counteract the difficulties which arise from vanishing and exploding gradients in very deep neural networks. Understanding that TD learning is not stochastic gradient descent, and has no direct relation to deep artificial neural networks, we can then ask whether such problems apply to linear TD problems, such as the ones explored in our paper. The problems of applying these optimizers to TD learning is demonstrated in part in the empirical comparisons done by , who provided a comparison of RMSPROP and TIDBD in their Figure 11; their analysis showed RMSPROP performed worse than TIDBD on their TD-learning task. RMSProp produced empirical return errors so high, that it could not even be plotted alongside the TD-specific step-size adaptation methods.

By implementing an individual step size for each input to a learning agent, it is possible for an agent to treat different inputs differently during learning. One extension of scalar step-size adaptation methods to a non-scalar form is Temporal-Difference Incremental Delta-Bar-Delta (TIDBD) . In their introduction of TIDBD, investigated adaptation of vector step sizes on a feature level, comparing how TIDBD adapts the step sizes for noisy features versus prediction-relevant features. In this work, we translate TIDBD to a more realistic setting. Rather than investigating deteriorating features, we investigate deteriorating sensors; we consider the case where a set of sensors freezes or becomes noise, preventing perception of a useful signal. Such a situation rarely translates cleanly to a simple set of unrelated feature noise in the feature representation. found in their experiments that TIDBD could outperform TD methods that lack step-size adaptation. As an extension of the work done by , we compare TIDBD against TD methods on a robot data stream with far more parallel signals than any prior test domains. In addition, we consider the viability of TIDBD on this complex data in terms of computation and memory.

A meta-learning method that can perform comparably to, or outperform, classic TD, yet avoid the need for time- and labour-intensive parameter tuning, is one main component for making predictive architectures practical in real-world applications. Although meta-learning methods promise to adapt parameters without human intervention, they themselves introduce new meta-parameters. Fortunately, the algorithmic performance is more robust with respect to these meta-parameters, as shown by This robustness allows these meta-parameter to be set with default values, rendering a parameter search unnecessary.

As main contributions of this work, we provide deeper understanding and intuition about the effect that using TIDBD will have on prediction-learning tasks involving complex, real-world data. We furthermore investigate the robustness of TIDBD on our robotic data set in order to reaffirm previous experiments. In what follows, we demonstrate how TIDBD adapts the step sizes in TD learning when confronted with a non-stationary environment. By examining the operation of TIDBD in comparison to classic TD and its ability to perform feature selection in relation to specific signals in the robotic arm, this work carves out insight that will help others design persistent agents capable of long-term autonomous operation and learning.

# Prediction-Learning Methods

Key to the construction of predictive knowledge systems is the way predictions are specified. One proposal is to express world knowledge as a collection of General Value Functions (GVFs) . Interaction with the world is described sequentially, where at each time step $t$, an agent takes an action $A_t \in \mathscr{A}$, which causes a transition from $S_t$ to $S_{t+1} \in \mathscr{S}$ assumed to be specified by a Markov Decision Process. The agent's choice of action, $A_t$, is determined by a probability function $\pi: \mathscr{S} \times \mathscr{A} \longrightarrow [0,1]$, known as a *policy*. We model our world by forming predictive questions about our sensations, which we phrase as GVFs—predictions about a signal of interest, or *cumulant*, $C$, from the environment over some time-scale or horizon defined by $\gamma \geq 0$, and some behaviour policy $\pi$. The discounted future sum of the cumulant is known as the *return*, $G_t = \sum^{\infty}_{k=0}\gamma^k C_{t+k+1}$. A GVF, $V$, is the expected return of this cumulant: $V(s;\pi,\gamma,C) = \mathbb{E}[G_t | S_t = s]$, which can be estimated using incremental online learning methods, such as TD learning . A collection of GVF learners is called a *Horde* .

In complex domains, such as the robotics domain we explore in this paper, the state space can be large or infinite: we must use function approximation to represent GVFs. The most common form of function approximation that has been used in fundamental work with GVFs is linear function approximation . Furthermore, there are multiple demonstrations showing the usability of linear function approximation in combination with GVFs in different applications, such as laser welding and prosthetic limbs . For these reasons, we use a linear function approximation approach in this work. To construct our features, we use selective Kanerva coding, a method shown by to be less sensitive to the curse of dimensionality than tile coding, yet still offering linear complexity. As our state space has $108$ dimensions, this is an important advantage. Selective Kanerva coding represents the state space with a number of *prototypes*. These prototypes are points in the state space which are typically distributed throughout the state space based on a uniform random distribution. In the vanilla Kanerva coding algorithm, a state would be represented by all prototypes within a defined radius around a state. This leads to a variable number of active prototypes, resulting in high-variance updates for the value function. However, in *selective* Kanerva coding, a state (point in this space) is represented by a constant number of the closest prototypes, providing a binary feature vector indexed by the prototypes. The constant number of active prototypes leads to less variance in the magnitude of the update.

When performing TD learning with linear function approximation, we estimate the value $V(s)$ as the dot product of a weight vector $\textbf{w}$ with a feature vector $\textbf{x}(s)$, constructed through selective Kanerva coding to act as the state representation of $s$. We improve our estimate $\textbf{w}^\top\textbf{x}(s)$ of $V(s)$ through gradient descent in the direction of a TD error $\delta_t = C_{t+1} + \gamma V(S_{t+1}) - V(S_t)$ (Algorithm , Line 3). The weights, $\textbf{w}$, for each GVF learner are updated on each time step.

## AutoStep TIDBD

When learning GVFs, two perennial challenges are setting parameters and choosing adequate representations. Oftentimes a representation is chosen as a fixed set of features, and this set of features and appropriate parameter settings are selected based on the results of extensive sweeps over large datasets. In the online lifelong continual learning setting, such a sweep is an impossible task. When we expect learning to occur over the life of the agent in non-stationary environments, we cannot ensure that a setting appropriate for only a subsample of experience will be appropriate in all situations.

This difficulty of setting the parameters for a single GVF is further compounded when we expect many predictions to be learned, as is the case in predictive knowledge architectures. In particular, no single step size (also known as *learning rate*) will be appropriate for every prediction in a Horde, and no single feature representation will be appropriate for every prediction in a Horde. As we cannot identify the best representations and parameters beforehand, it would be ideal to be able to adapt them through experience. To this end, we explore TIDBD: a meta-learning method for temporal-difference learning that adjusts the step-size parameter on a per-feature basis. By adapting step sizes through stochastic meta-descent, TIDBD not only provides a method of tuning parameters, but also a form of representation learning.

When hand-tuning each GVF, a single appropriate step size $\alpha > 0$ is typically chosen, resulting in the use of the same step size for every feature. TIDBD, however, adjusts a vector of many step sizes—one step size for each feature. For a binary feature vector $\textbf{x}(s) \in \mathbb{R}^n$, for example, there would be a corresponding vector of learned step sizes $\boldsymbol\alpha \in \mathbb{R}^n$, where $n$ is the number of features. At its core, TIDBD is performing stochastic meta-descent to modify the step-size parameters to minimize the squared TD error. This meta-descent results in step-size adaptation based on each feature's relevance; features which are highly correlated to the prediction problem should be given large step sizes, while irrelevant features should contribute less to updates and be given smaller step sizes.

TIDBD can be thought of as having two components 1) IDBD-based meta-learning and 2) AutoStep-based normalization. IDBD-based meta-learning provides the updates of each step size $\alpha_i$ by learning the meta weights $\boldsymbol\beta$ through meta-descent (see update in Algorithm , line 10). Intuitively, this means that the more correlated recent weight updates are for a given feature $x_i$, the more effective it is to make a single large update, and thus the greater the value of the step size $\alpha_i$. This correlation is tracked using the decaying trace of recent weight updates, $\textbf{h}$ (updated in line 17). For more intuition on how IDBD abstracts the problem of tuning, see Section 2 (IDBD) of .

AutoStep-based normalization $\xi$ is a an additional factor for the meta weight update (see line 10). By adding AutoStep-based normalization, we 1) ensure that the effective step size is not so large that we overshoot on a given update for a particular example (line 11), and 2) maintain a running average of recent weight updates to ensure that the step sizes do not diverge if many successive weight updates are correlated (lines 6-9). The variable $\xi$ acts as a weight-update normalizer: we take the maximum value between the absolute value of the most recent weight update (line 7), and a decaying trace of recent updates (line 8), where $\tau$ is a parameter that determines the rate of decay. We take a decaying trace of $\xi$, as it ensures the normalizer can recover gracefully from outliers and extreme values.

While a common criticism of meta-learning is the introduction of additional meta parameters, TIDBD is insensitive to both of its meta parameters: the meta step size, $\theta$, and the rate-of-decay parameter, $\tau$. suggest $\theta = 10^{-2}$ to be a good choice over a variety of different prediction problems . To confirm this suggestion, we performed a robustness study, described in Section . As shown by , the rate-of-decay parameter $\tau$ does not have a significant influence on performance. We therefore set it to the suggested value of $\tau=10^4$.

# Experimental Setup

We gathered the data for our experiments from the Modular Prosthetic Limb (MPL v3) —a state-of-the-art bionic limb, seen in Figure , which is capable of human-like movements. The MPL includes $26$ articulated joints in its shoulder, elbow, wrist, and hand. It provides $17$ degrees of freedom. Each motor has sensors for load, position, temperature, and current; each fingertip is outfitted with a $3$-axis accelerometer and with $14$ pressure pad sensor arrays. Together, these provide a data stream of $108$ real-valued sensor readings that is shown in Figure .

Experiments by suggest that real-time prediction learning can make the control of artificial limbs more intuitive for the user. In particular, anticipation and adaptation are highly important given the world and tasks encountered by a prosthetic limb are continuously changing. Therefore, the arm is an interesting showpiece as an autonomous learner .

In this work, the sensor readings acted as both the cumulants for predictions that the system was tasked with learning and as the state information provided to the system. Since our data included $108$ sensor readings, the architecture of the system was a Horde of $108$ predictions (general value functions), with each prediction estimating the expected discounted return for a different sensor reading. For each prediction, we used a discount rate of $\gamma = 0.9$. This discount rate can be thought of as resulting in a 10-step time scale, because it refers to, in expectation, the sum of the cumulant over 10 time steps. Succeeding time steps were $0.265$ seconds apart, on average, so a $10$-step time scale refers to $2.65$ seconds. This value is potentially well suited to capturing some comparatively slow movements, e.g., the elbow extension or flexion, but might result in averaging over very fast movements, e.g., a fast grasping movement. As the computations for the predictions and ideal return were undertaken offline—though the computations for the predictions used online methods—the computation time did not affect the length of the time step.

As state information to update each prediction, the learning system has access to all $108$ sensor readings at the current time step. These sensor readings are normalized and fed into the selective Kanerva coder to obtain feature vector $\textbf{x}(s)$. For intuition on the Kanerva coder, one can think of a $108$-dimensional cube, with each side being the $\left[ 0, 1 \right)$ interval. There are a constant number, $n$, of fixed prototypes (points) within the cube. Each normalized value fed into the Kanerva coder is a point in the cube, and can be represented by the $\eta$ nearest prototypes. We call this set of nearest prototypes *active*.

Note that both classic TD and TIDBD are online methods for estimating general value functions, meaning they predict the expected return based on only information observed up to the time step they are estimating the value for. They use previous estimates, the current cumulant, and the current feature vector to make an educated guess about the future of the cumulant signal: the discounted return. A perfect estimate of the return could only be made with cumulant readings that have yet to be made.

We measured the performance of TIDBD through a comparison with classic TD. In particular, we considered the root mean squared error (RMSE), which is essentially a measurement of the difference between the true return and the value (the expected value of the return). We computed the RMSE over all predictions for a single time step, $t$, as follows. $$\textit{RMSE}_t = \sqrt{\frac{\sum_{i=1}^{108}\left( \frac{G_t^{(i)} - \textbf{x}(s_t)^\top\textbf{w}_t^{(i)}}{\left\lvert G_t^{(i)}\right\rvert} \right)^2}{108}}$$ The superscript $(i)$ denotes association with the $i$th prediction of 108. Normalization (division of the return and value estimate by ${\scriptstyle \left\lvert G_t^{(i)}\right\rvert}$) was done to make the RMSE meaningful, as the returns (and associated predictions) for different sensors were on different scales. Note that it would be unlikely for the RMSE to reach zero. While the return was computed taking all sensor readings for the full experiment into account, both classic TD and TIDBD used only sensor readings up to the present time step. These observations do not provide enough information to perfectly predict the future.

For a meaningful comparison of classic TD with TIDBD, the parameters needed to be carefully tuned. The best parameters were chosen based on minimizing the RMSE over all predictions, summed over all time steps of the experiment. We therefore performed parameter sweeps for the number $n$ of prototypes in the Kanerva coder, the ratio $\eta$ of active prototypes to the total number of prototypes, and for the scalar step sizes $\alpha$ for each prediction for classic TD. The candidates for each parameter are shown in Table . The candidates for $n$, the number of prototypes, were chosen based on the recommendations provided by . We used a full factorial experimental setup, resulting in $264$ different parameter settings for the experiments with a fixed step size, those for classic TD. Because the TIDBD experiments did not require a sweep over potential step sizes $\alpha$, there were only $24$ different parameter settings for the TIDBD experiments (accounting for candidates for $\eta$ and $n$). In total, we conducted $288$ different experiments for our comparison of TD learning and TIDBD.

| Parameter | Count | Candidates |
|:--:|:--:|:---|
| $n$ | 4 | 10000, 20000, 30000, 40000 |
| $\eta$ | 6 | 0.001, 0.002, 0.004, 0.008, 0.016, 0.032 |
| $\alpha$ | 11 | $\frac{0.001}{n \cdot \eta}$, $\frac{0.002}{n \cdot \eta}$, $\frac{0.004}{n \cdot \eta}$, $\frac{0.008}{n \cdot \eta}$, $\frac{0.016}{n \cdot \eta}$, $\frac{0.032}{n \cdot \eta}$, $\frac{0.064}{n \cdot \eta}$, $\frac{0.128}{n \cdot \eta}$, $\frac{0.256}{n \cdot \eta}$, $\frac{0.512}{n \cdot \eta}$, $\frac{1.024}{n \cdot \eta}$ |

Parameter candidates tested in full factorial design.

To set the parameters for classic TD, the experiment was first run with a fixed step size shared by all GVFs in the Horde. In these experiments, using $n = 30000$ and $\eta = 0.032$ yielded the lowest RMSE in comparison to other Kanerva coder parameter choices, regardless of the choice of step size. In a second step, the RMSE for each GVF was calculated for each step size candidate, so the best step size for each GVF could be chosen independently. The best step sizes ranged from $\frac{0.001}{n\cdot\eta}$ to $\frac{0.256}{n\cdot\eta}$, where the product $n\cdot\eta$ is the number of active features.

The parameters that yielded the best performance in terms of RMSE for classic TD also performed best in the parameter sweep for TIDBD. For each feature, the step size was initialized to $0.00104$, which corresponds to an initial value of $\frac{1}{n\cdot\eta}$. After the best parameters were established for classic TD and TIDBD, 30 independent trials were performed for each.

We programmed the robotic arm with a repeating series of periods of rest and periods of motion. The experiments started with the arm holding its position for five minutes. This period of rest was followed by five minutes of the arm repeating a complex pattern of movement that was programmed using a data glove. The movement pattern included motion of all joints and involved movements that humans with intact arms take for granted, like grasping or flexing one finger after another. For a better understanding, the exact movement pattern can be found online at [`https://blinclab.ca/mpl_teleop_video/`](https://blinclab.ca/mpl_teleop_video/). The movement pattern was $100$ seconds long, so was repeated three times during the five-minute period of movement. The periods of rest and movement alternated three times, totalling $30$ minutes.

During the rest period, each position, velocity and load sensor would be expected to report a constant signal, up to machine precision. Such sensor values should be easy to learn. During the movement pattern, on the other hand, the robot is in contact with human intention, so the predictions become far more difficult to make. The full series of periods of rest and movement provided an interesting test case, approximating intermittent stationarity and non-stationarity, to investigate the effect of TIDBD on GVF predictions.

Beyond our investigation of TIDBD with all sensors fully functioning, we also investigated how TIDBD reacts when confronted with two commonly occurring sensor failures: 1) sensors being stuck and 2) sensors being broken. A stuck sensor typically outputs a constant signal with a small amount of sensor noise , while a broken sensor typically outputs Gaussian noise with a high variance . In both experiments, the signals from all four sensors in the elbow were replaced: in the first, with Gaussian noise of $\mathcal{N}(1,0.5)$ for the stuck sensors, and with Gaussian noise of $\mathcal{N}(0,10)$ for the broken sensors.

Lastly, to investigate the robustness of TIDBD with respect to the initial step sizes $\alpha$ and its meta step size $\theta$, we performed sweeps over the data set with different values for the initial step sizes from Table within the same range as for TD and with different meta step sizes in the range of $\theta = \{0.005, 0.01, 0.02, 0.04, 0.08, 0.16\}$.

# Results and Discussion

The experiments were designed to investigate the effect that TIDBD has on predictions about the signals provided by a sensor-rich robotic arm. As a baseline, classic TD with an extensive parameter search was implemented. Three different scenarios were introduced: the predictions for different patterns of movement and rest, the predictions for the same patterns when the four elbow sensors are stuck and report a slightly noisy constant signal, and the predictions for the patterns when the four elbow sensors are broken and only report noise.

## Comparison of Classic TD and TIDBD

| GVF setting | Fixed step size |  TIDBD  |
|:-----------:|:---------------:|:-------:|
|   Rest 1    |     1531.84     | 1222.90 |
|   Rest 2    |     983.72      | 1009.14 |
|   Rest 3    |     1023,00     | 1046.66 |
| Movement 1  |     1105.32     | 1323.94 |
| Movement 2  |     1085.81     | 1184.09 |
| Movement 3  |     1003.21     | 1072.13 |

Average RMSE over 30 independent runs

We first consider the root mean squared error (RMSE, as defined in Section ) for both classic TD and TIDBD in our initial experiment, where all sensors are fully functional. The top pane of Figure shows the RMSE for each period of rest and movement. It can be seen that the highest error for both classic TD and TIDBD occurs during the first period (Rest 1). This can easily be explained by all GVFs being initialized without any knowledge about the sensor readings—the RMSE for the first time steps will therefore be high. These high errors can be well seen in the violin plots, in the middle and bottom panes, which show the distribution over the errors the predictions made, the extrema, and the medians in the subplots for both classic TD and TIDBD. The maxima for TD and TIDBD were considerably higher in this period than for any other part of the experiment and the error distribution was much broader, as indicated by the colored area in the violin plots in Figure . Unsurprisingly, TIDBD exhibited a higher RMSE than classic TD at the beginning of the experiment, as its step sizes were initialized more aggressively and were not tuned to the predictive task.

The error for the second rest period was already considerably lower. Perhaps unintuitively, the error for the third rest period increased again. This can be explained by the sensor data from Figure . One of the load sensors started to drift in the third rest period. As this pattern had not been seen in any of the rest periods before, the RMSE peaked again—the pattern of a drifting sensor had not been learned, yet. For the periods of movement, a steady decrease in RMSE was observed.

On average, TIDBD had a slightly higher RMSE for the $30$-minute experiment. The exact errors for each period of rest and movement can be found in Table . It is important to recognize that our parameter sweep over step sizes provided an advantage to classic TD; because the step sizes were chosen to minimize the RMSE for the full experimental data, their choice inherently provided some information, which TIDBD did not receive, about the data. In a real-world application, providing this advantageous information in the form of parameters would not be possible, as the learner would be constantly faced with new, unknown data after the parameters have been set.

Despite this advantage, TIDBD and classic TD performed comparably with respect to the RMSE. This result indicates that TIDBD can act as an alternative to tuned classic TD learning, without the time- and labour-intensive setup that TD learning requires for tuning.

The choice of parameters appears to have a tremendous impact on the learning performance. Wrong parameters might result in almost no learning at all or constant overshooting. Adapting the parameters based on the incoming data should therefore result in better and more steady performance. To see whether TIDBD demonstrates more steady performance, we considered the sensitivity of the RMSE to the parameter settings for each algorithm. Our experimental data shows that classic TD was indeed strongly dependent on the learning rate: the standard deviation of the RMSE over the $264$ combinations of parameters in our full factorial design was $\sigma_{\textit{TD}\_264} = 43,734.46$. In comparison, once the best step sizes for classic TD were preselected, the standard deviation for the remaining $24$ experiments was $\sigma_{\textit{TD}\_24} = 313.42$. This value is over $100$ times smaller than the standard deviation for all $264$ experiments.

TIDBD, for which there are no learning rates to tune, attained a standard deviation of $\sigma_\textit{TIDBD} = 1,507.24$ over the $24$ Kanerva coder parameters. This value is ${\sim}30$ times smaller than $\sigma_{\textit{TD}\_264}$, but ${\sim}5$ larger than $\sigma_{\textit{TD}\_24}$. The difference in the standard deviation between classic TD with a preselected learning rate and TIDBD is most likely due to the duration of the experiment. As classic TD was initialized with optimized step sizes, it was able to perform more effectively over a short period of time.

The above comparison of classic TD and TIDBD in terms of RMSE is valuable because it helps us understand the performance of TIDBD and demonstrates its potential for attaining similar performance to classic TD without a manual tuning process for the learning rate. However, feasibility also depends on computation and memory, and there is an associated cost with using TIDBD to update the step sizes without human interaction. For each weight in the $108$ GVFs, an additional step size was required. Given a feature representation with $30,000$ features per GVF, $3,240,000$ step sizes were required in this particular setting. Per GVF, three additional vectors of the same size as the number of features are required. In our Python implementation, each of the three additional weight vectors required $0.24$ megabytes, totalling to an additional $0.72$ megabytes. The additional computation for updating this larger number of step sizes increased the time for updating all GVFs from $0.025$ seconds to $0.28$ seconds. However, as this corresponds to nearly four updates per second, it was still within the requirement for a prosthetic limb.

The computations were performed using a Linux Mint $18.3$ OS system with an i$7$-$7700$HQ CPU with a $3.80$GHz clock rate, $6$ MB of shared L$3$ cache and $32$GB DDR$4$ RAM. With the ongoing evolution of hardware, we expect it to become possible to maintain and update even greater numbers of GVFs or to reduce the time needed for computation.

Our experimental data also offers us the opportunity to gain insight into the meta-learning process resulting from applying TIDBD. TIDBD assigns different step sizes to different GVFs and different features. As a result, different features contribute different amounts over time. Step sizes that are related to unimportant or noisy features will be reduced. These individual updates can be interpreted as a feature selection mechanism—TIDBD actively adapts its representation of the predictive problem, solely based on interactions with the environment .

To better understand how TIDBD changes step sizes throughout the experiment, Figure shows four snapshots of the distribution of the step sizes . In each subplot, the orange bar shows step sizes that had not yet been updated, due to the corresponding features not being activated; the blue bars represent the step sizes that had been updated by TIDBD. Subplot (a) shows the step sizes at initialization. All of the step sizes were initialized to $0.00104$. As we would expect, Subplots (b), (c) and (d) show that the longer the experiment had run, the more the step sizes had spread out. Subplot (d) shows that the step sizes were set, by the end of the experiment, to within the range from $\num{8.0008e-05}$ to $0.00255$.

Although TIDBD actively improves its representation by adapting the step sizes, it was still sensitive to the representation that was provided, as its performance considerably varied ($\sigma_\textit{TIDBD} = 1,507.24$) with the Kanerva coder parameters—information that is not provided to the learner due to an insufficient state representation cannot be compensated for. Within the realm of robotics, the state representation is often negatively impacted by damage to the sensors. We explore this problem in the following section by comparing the behaviour of TIDBD on data with simulated broken and stuck sensors with the behaviour demonstrated in the first experiment, shown in Figure .

## Stuck Sensors

For the second experiment, recall that the elbow sensors were replaced with low-variance Gaussian noise, $\mathcal{N}(1,0.5)$, to simulate them being stuck. The distribution of adapted step sizes at the end of this experiment can be found in Figure .

In comparing Figure with Figure , of particular note is the fact that, with simulated stuck senors, some step sizes were adapted to be much larger than any adapted during normal operation; the maximal step size when all sensors were functioning was $0.00255$, while Figure shows step sizes of up to $0.005$, approximately twice as large.

The step sizes for both the predictions with stuck sensor signals as their cumulants *and* for the remaining "unaffected" predictions increased in magnitude. This result may be counterintuitive at first. For a constant signal with a small amount of noise, we would expect the step sizes to decrease, as such a signal does not contain a significant amount of information. In the setting at hand, this reaction is countered by the choice of representation. As the Kanerva coder prototypes were randomly distributed in space, the small amount of noise could actually be expected to constantly lead to different prototypes being activated. At the same time, the cumulants were staying nearly constant, due to the variance being small. This discrepancy between almost stationary cumulants and a changing representation appears to have lead to increasing step sizes, as TIDBD tried to achieve the necessary updates in fewer steps. Each feature was assigned a higher value, likely due to these updates being distributed over a wider range of features, resulting in higher step sizes. While these increasing step sizes did not necessarily improve the representation, they are clearly distinguishable from step sizes that occurred during the normal functioning of the robotic arm, thus providing important knowledge about the sensor failure.

## Broken Sensors

The problem of broken sensors is a common one in robotics and of high interest in long-term autonomous systems. For the final experiment, the four elbow sensors were replaced with Gaussian noise, $\mathcal{N}(0,10)$, which corresponds to broken sensors that output noise. Such broken sensors do not contain meaningful information, as their output will be purely random—we therefore expect TIDBD to decrease the corresponding step sizes for these sensors.

Figure shows the step size distribution for the experiment with broken sensors that output high-variance noise drawn from $\mathcal{N}(0,10)$. Subplot (a) depicts the step-size distribution for the four sensors that output noise. The maximum step size is only $0.0017$. The step sizes observed during this experiment were considerably smaller than they were in the experiments where all sensors function well. The average step size for the broken sensors is $0.00037$, while the average step size for these four sensors in the experiment with functioning sensors is $0.00065$. Subplot (b) shows the distribution for the remaining $104$ sensors. While the maximum in this experiment, with a value of $0.0028$, was almost identical to the maximum of $0.0025$ in the experiment where all sensors work well, there is a considerably distinction in the average step sizes. For the experiment with broken sensors, the average step size was $0.0006$, while it was $0.00077$ in the experiment where all of the sensors are functioning as expected. The RMSE for the $104$ functioning sensors, given broken elbow sensors, was calculated for both a TIDBD Horde and a classic TD Horde. The information provided by the elbow sensors was used in the feature representation $\textbf{x}(s)$, but since these sensors are broken, they only provided irrelevant, distracting information to the predictors. For the classic TD Horde, the RMSE for the $104$ functioning sensors increased to $1,315,850.16$. Step-size adaptation using TIDBD resulted in a considerably lower RMSE of $509,220.75$ in this experiment.

As expected, the step sizes corresponding to the four sensors that were replaced by noise were considerably decreased when compared to the step sizes during normal operation. Based on the interaction with these sensors, TIDBD appears to decide that it cannot learn additional information about them and to exclude them from further learning. The step sizes for the remaining $104$ sensors remained almost the same as in the normal operation of the arm. However, the distribution of step sizes in the intact sensors changed slightly as more step sizes were decreased in value—potentially to exclude features that correspond to the noisy inputs from impacting the predictions about the functioning sensor values. The RMSE for the remaining $104$ sensors supports this intuition, as it is ${\sim}2.5$ times lower ($1,315,850.16$ for classic TD vs. $509,220.75$ for TIDBD) for TIDBD than for classic TD.

## Parameter Sensitivity for TD and TIDBD

Using TIDBD to automatically adapt parameters requires the use of meta-parameters. To investigate the sensitivity of TIDBD with respect to initial step sizes $\alpha$ and the newly introduced meta step size $\theta$, we performed sweeps over different initial values of step sizes and different meta step sizes. The results are shown in Figure . Shown are the accumulated RMSEs for the whole experiment for TD and TIDBD with different values for the initial step sizes. As expected, the performance for classic TD resembles a bowl, clearly visible in the second plot. It performs poorly for very small step sizes and very large step sizes. Between these two extrema, there is a small window of good performance that traditionally has to be found by a parameter sweep.

TIDBD, however, shows much more robust performance with respect to initial parameters. While the performance is more steady over different parameters, there is still a difference for different meta-step sizes. Although this is notable, the most important takeaway from this plot is the big gap in performance between TIDBD (with any meta step size) and classic TD. The bowl shape for TIDBD is an artifact of the experiment, not of the algorithm. The experiment has a finite amount of data which results in theoretically ideal learning rates, minimizing the error for this finite data. The closer the initial learning rates are to the ideal learning rates, the less TIDBD has to adapt the learning rates to match the ideal learning rates. This will result in a smaller error. For this reason, it makes sense that a step size adaptation algorithm with a higher meta-step size ($\theta=0.16$) will perform better on a limited data set, as it can adapt the learning rates in fewer times steps.

More importantly, the steadier performance for $\theta=0.16$ does not mean better performance when it comes to long-term error. In fact, TIDBD with lower meta step sizes demonstrates lower error for some initial step sizes. This may indicate that the meta step size $\theta=0.16$ is in fact too aggressive to converge. Therefore, the key message is the big difference between TIDBD (with any meta-step size) and classic TD. These plots confirm the results of and show that the meta step size and the initial step sizes can be used as default parameters within a reasonable range.

Together, the results in this paper not only support the usability of TIDBD to independently learn and update step sizes for predictions without the need of human assistance, but furthermore to independently adapt the representation that is used for a predictive knowledge approach. As TIDBD updates the step sizes based solely on interactions with the environment and is grounded in the observations that are received from said environment, it can truly function on its own—even when implemented in a long-lived application.

# Conclusion

The experiments in this paper were conducted to investigate TIDBD—a step-size adaptation algorithm that assigns individual step sizes on the feature level. Four different experiments were performed on a sensor-rich robotic arm to gain further insight into the functioning of TIDBD. All four experiments utilize the data from alternating patterns of rest and movement. These four experiments result in three contributions: First, we demonstrate TIDBD to be a practical alternative to an extensive step size parameter search. Secondly, we show how TIDBD can be used to detect and characterize common sensor failures. As a third contribution, we explore TIDBD's sensitivity to its meta step size and to its initial step sizes in comparison to classic TD.

**First experiment:** We compared the predictive performance of classic TD with an extensive parameter search to the predictive performance of TIDBD. The additional computation required by TIDBD was still within our requirements for real-time computation and the memory used for TIDBD is negligible on modern systems. The results show that TIDBD and classic TD performed comparably in terms of the root mean squared error (RMSE). Although there is a set of fixed step sizes for which classic TD exhibits slightly less error on our data set, we expect TIDBD to perform better when applied in a lifelong learning setting. We expect the performance of TIDBD to exceed that of classic TD in long-term settings due to TIDBD exhibiting more improvement over time than classic TD in our experiments. These results therefore suggest that an extensive learning rate parameter search is needless.

**Second experiment:** We then explored the changes in the learning rates with several stuck sensors. The changes in the TIDBD step sizes were clearly distinguishable from changes seen during normal functioning of the arm (as explored in the first experiment), therefore providing an indicator to detect this type of sensor failure.

**Third experiment:** We replaced several sensors with high variance noise, simulating broken sensors. TIDBD decreased the step sizes corresponding to the broken sensors, which resulted in these inputs being gradually excluded from the updates—it automatically learned the unimportance of these inputs.

**Fourth experiment:** We investigate the performance in terms of accumulated RMSE for TD and TIDBD when initialized with step sizes of different magnitudes. Furthermore, the performance of TIDBD is evaluated for different meta step sizes. While the performance for TD shows a huge dependence on the initial step size value, TIDBD is more robust towards these initial step sizes and its meta step size. This shows that the usage of TIDBD is more robust with respect to its initialization, making it a viable alternative to an extensive parameter search.

These four results—the permanent updates of step sizes to accommodate non-stationarity, the distinct reaction to stuck sensors, the automatic feature selection for uninformative sensors and the robustness with respect to its initialization—are promising key features for long-term autonomous agents. They empower an agent to not only adapt its learning based on interactions with its environment, but to evaluate and improve its own perception of said environment.

Furthermore, as the step sizes contain information about the past for each feature, they can provide an important source of information to the agent itself to learn from. As has been argued prior to this work , these introspective signals provide a helpful source of information to enable an agent to better understand its environment and its own functioning within its environment

The insights presented in this paper provide deeper understanding and intuition about the effects of TIDBD, aiming to help other designers in creating agents that are capable of autonomous learning and adaptation through interaction with their environment.

# Acknowledgements

This research was undertaken, in part, thanks to funding from the Canada Research Chairs program, the Canada Foundation for Innovation, the Alberta Machine Intelligence Institute, Alberta Innovates, and the Natural Sciences and Engineering Research Council. The authors also thank Jaden Travnik for discussions on Kanerva coding and Craig Sherstan as well as Richard Sutton for suggestions and helpful discussions.