# A Model for Evolution of Investors Behavior in Stock Market Based on Reinforcement Learning in Network

Authors: Xiaqun Liu; Yaming Zhuang; Jinsheng Li
Journal: Complexity
 (2020)

---

## Abstract

This paper builds an evolution model of investors behavior based on the reinforcement learning in multiplex networks. Due to the heterogeneity of learning characteristics of bounded rational investors in investment decisions, we consider, respectively, the evolution mechanism of individual investors and institutional investors on the complex network theory and reinforcement learning theory. We perform mathematical analysis and simulation to further explain the evolution characteristics of investors behavior. The conclusions are drawn as follows: First, the intensity of returns competition among institutional investors and the forgetting effect both have an impact on the equilibrium of their evolution as to all institutional investors and individual investors. Second, the network topology significantly affects the behavioral evolution of individual investors compared with institutional investors.


---

## Body

## 1. Introduction

There are significant uncertainty and unpredictability in the stock market due to the complicated factors including investors’ bounded rationality and policy correlation effects. Investors tend to frequently trade and blindly follow the volatility of stock market, which can easily react upon the market exacerbating the instability [1–4]. Therefore, it is of great significance to study the evolution of investor behavior in the stock market. The research is more effective in grasping the behavior evolvement rule of stock market.In recent years, reinforcement learning has become a cutting-edge application in the financial field with the development of big data and the increasing demand for financial data analysis. At present, many literatures have applied reinforcement learning to the research on high-frequency trading, investment portfolio, and other fields. Jangmin et al. [5] proposed a new stock trading method combining dynamic asset allocation and reinforcement learning. Shimokawa et al. [6] introduced nonlinear effects into the ordered TD-type learning model and constructed an augmented TD-type learning model to predict investors’ behavior. Bekiros [7] proposed an adaptive fuzzy Actor-Critic reinforcement learning and built a financial trading system based on it. Tan et al. [8] constructed an arbitrage-free algorithmic trading system with the adaptive network fuzzy inference system (ANFIS) based on reinforcement learning. Bertoluzzo and Corazza [9] applied Q-learning and Kernel-based RL algorithms to automated financial trading. Feuerriegel and Prendinger [10] developed an automatic decision-making system with the news-based sentiment data and price momentum based on the supervision and reinforcement learning. Pendharkar and Cusatis [11] constructed a pension investment portfolio based on reinforcement learning. Yang et al. [12] believed that investor sentiment is an important factor affecting market returns, so they designed a trading system based on investor sentiment reward by Gaussian inverse reinforcement learning.Network science theory is important to analyze investor behavior and it is a supplement to traditional statistics and experimental economics [13–15]. In recent years, many scholars have studied the investment behavior in the stock market from the perspective of network science. Existing research is divided into three aspects: analysis on investor network topology, empirical description of investment behavior, and research on investor behavior under the interaction mechanism.At present, some scholars have analyzed the topological structure of financial network through using actual financial data and found that these financial networks have the characteristics including disassortative architectures [16], small-world properties [17], power-law degree distribution [18–20], and community structure [21, 22]. Through the study on the network structure, some scholars further analyzed the investment behavior in the financial network. Long et al. [23] and Baumohl et al. [24] studied the volatility spillover effect of the stock market based on the stock network. Pareek [25] constructed a network model of mutual fund interconnection with holding the same stock as the “link” and revealed the inherent formation mechanism of fund herding behavior. Chung et al. [26] constructed an empirical investor network with the account transaction data of investors in the Taiwan Stock Exchange from 2005 to 2014 and analyzed the impact of information dissemination on stock returns.As the actual financial data is not completely available, methods such as theoretical analysis and simulation have become important choices for many scholars to study the investor behavior in financial network. Wang et al. [27] divided the investors into three types including experts, speculators, and followers, to study how the information quality of experts affects investor behavior in the social network. Bian et al. [28] constructed an evolutionary model of investor behavior in the stock market based on network coordination game and studied the impact of investor behavior on financial market. Bakker et al. [29] and Stefan et al. [30] established an evolution model of investors’ investment behavior based on the investor’s social network to analyze investor behavior and its impact on the stock market. Xu et al. [31] proposed a weighted game model in investor network and found that the investor behavior evolution in WS small-world network is more stable than that in BA scale-free network. Khashanah and Alsulaiman [32] analyzed the performance of their investment behavior on the scale-free network by taking four types of investors, basic strategy traders, momentum traders, adaptive strategy traders, and zero-intelligence traders, into consideration. Cohen et al. [33] studied the information transmission in the financial market by applying a social network. Krichene and El-Aroui [34] divided investors into informed and uninformed investors, constructed a scale-free social network model, and analyzed the information asymmetry and herd behavior in different stock markets. Wang and Pan [35] built an artificial stock market model based on the heterogeneous information strategies in a dynamic scale-free network and studied the evolution of investors’ information strategies.In recent years, with the development of network science theory, many scholars have changed from a single-layer network perspective to a multi-layer network perspective to study investor behavior in stock market. Li et al. [36] analyzed the volatility of the stock market and the herd behavior of investors by using a bipartite network model of stocks and traders. Gao [37] explored the impact of margin trading on stock price volatility based on a double layered network of stocks investors. D’arcangelis et al. [38] analyzed the investment style of Italian pension fund by using a bipartite network model. Paulin et al. [39] constructed a double layered network of funds and assets and analyzed the reasons for the stock market crash. Lu et al. [40] studied the “herd effect” caused by the pursuit of portfolio diversity based on a bipartite network of funds and stocks. Biondo et al. [41] built a double layered financial network model to simulate information dissemination and investor transactions in the market. Wang and Chen [42] constructed the transaction model with a heterogeneous information in a double layered social network by introducing the double-factor interaction function. Souza and Aste [43] established a multi-layer network model by using social media data and financial data to predict the future stock market structure.In conclusion, there are many existing literatures studying investor behavior in stock market with reinforcement learning theory, but these literatures are mainly based on a single-layer network model and a single reinforcement learning mechanism, ignoring the heterogeneity of investor network structure and investor learning. Therefore, this paper divides investors in the stock market into the individual ones and institutional ones. Then, we analyze the decision-making mechanism of individual investors and institutional investors using the complex network theory and reinforcement learning theory. Based on it, the evolution model of investors behavior in the multi-layer network is constructed. Finally, this paper is investigated by mean-field analyses and simulations on the evolution characteristics of investors behavior．


## 2. The Model

### 2.1. Evolutionary Mechanism of Investor Behavior

Due to factors such as information asymmetry, the professionalism of investors, and differences in investment goals [44, 45], bounded rational investors are less likely to make optimal decisions only by observing the current stock market conditions. Experience is the easiest way to judge the feasibility of investment decisions. Before making the next investment decision, an investor usually refers to the returns obtained in the past. Even if the environment has changed, it will also repeat the strategy with better returns in the previous period [46, 47]. Therefore, the experience-based reinforcement learning method will directly affect investors’ investment strategies. However, there are significant differences between individual investors and institutional investors. Compared with individual investors, better performance can bring more capital inflows, which helps institutional investors achieve their goals, so there is intense performance competition among institutional investors [48]. Therefore, this paper divides investors into institutional investors and individual investors, the evolution model of investment behavior under different reinforcement learning strategies in network is constructed, as shown in Figure 1.Figure 1
The evolution mechanism of stock market investor’s decision.Based on the above analyses, the decision-making mechanism plays a crucial role in the evolution of investor behavior. In view of the differences between individual investors and institutional investors in investment decision-making, the learning strategies of investment behavior evolution of institutional investors and individual investors are constructed, respectively.


### 2.2. Evolutionary Model of Investor Behavior

SupposeG=N,E as the stock market, where N=1,2,…,i,…,n represents the investor and i,j∈E means the connection between investor i and investor j. ki refers to the degree of node i; pk represents degree distribution. Meanwhile, investors in the stock market are divided into institutional investors and individual investors, and the proportion of institutional investors in all investors is η0<η<1, so the proportion of individual investors is 1−η. In addition, it is assumed that there are three investment behavior states in stock market: ai∈−1,0,1, where ai=−1 means “sell,” ai=0 means “hold,” and ai=1 means “buy.” Based on this, the state of all investors in the stock market can be expressed as At=a1,t,a2,t,…,ai,t,…,aN,t at any time.mi0≤mi≤ki is set to indicate the number of individual investors which are associated with investor i, and ni0≤ni≤ki is set to indicate the number of institutional investors which are connected with investor i; then mi+ni=ki. Among them, it can be assumed that there are ni,1 buyers, ni,−1 sellers, and ni,0 holders of institutional investors associated with investor i, and ni,1+ni,−1+ni,0=ni. Similarly, it can be assumed that, among the individual investors associated with investor i, there are mi,1 buyers, mi,−1 sellers, and mi,0 holders, and mi,1+mi,−1+mi,0=mi.


#### 2.2.1. Evolution Rules of Individual Investors Behavior

①Individual investors reinforcement learning strategies: Strahilevitz et al. [49] found that investors tend to repurchase the stocks they previously sold for a gain while ignoring the stocks they previously sold for a loss. Linnainmaa [50] found that household trading intensity depends on past performance. In the stock market, the investment performance of individual investors is largely affected by personal experience. In particular, investors tend to pay more attention to some successful experiences, even if such experience cannot be replicated in the future [51, 52]. These findings can be explained by reinforcement learning models. Individual investors will adjust their behavior according to the changes in the stock market at any time. On perceiving the changes of the market, they will take corresponding actions, such as buying, selling, or holding. The market will give certain feedback based on actions. If the action is successful, the probability of it being selected will increase [53]. Kaustia and Knüpfer [46] found that, compared with Bayesian learning theory, reinforcement learning theory can better explain the positive correlation between investors’ past IPO returns and future subscriptions. Choi et al. [54] used reinforcement learning theory to explain the savings behavior of individual investors. Erev and Roth [55] established a reinforcement learning model, which can well describe the evolution of individual behavior in economic experiments. Therefore, this paper refers to Erev and Roth’s reinforcement learning model to fit the evolution rules of investment behavior of individual investors.Supposeqi,at represents the attractiveness of strategy a to investor i at time t. Forgetting parameter λ is used to gradually weaken the influence of past experience, and λ∈0,1 always holds. ξit represents the return of investor i at time t. When the decision of investor i is the same as the optimal decision max∑kj∈Sikjaj,1,∑kj∈Sikjaj,−1,∑kj∈Sikjaj,0, his/her returns ξi=1. Otherwise, ξi=−1. The investor’s strategy update rules are as follows:(1)qi,at+1=11+e−λe−λqi,at+Ela,ξ,(2)Ela,ξ=ξ,a=l,0,a≠l.②Individual investment decisions: based on the above analyses, the probability that individual investori chooses to “buy” is pIait=1=qi,1t/∑aqi,at. The probability that individual investor i chooses to “sell” is pIait=−1=qi,−1t/∑aqi,at. The probability that individual investor chooses to “hold” is pIait=0=1−pait=1−pait=−1.


#### 2.2.2. Evolution Rules of Institutional Investors Behavior

Greenwood and Nagel [56] analyzed the performance of experienced and inexperienced fund managers during the bubble. They found that inexperienced managers would buy technology stocks in large quantities during the run-up and sell them during the downturn. Hence, similar to individual investors, institutional investors also rely on reinforcement learning in trading. The historical performance of institutional investors will influence the capital flow and investment choice of individual investors. In order to maximize profits, there is a competitive relationship among institutional investors [48]. Compared with individual investors, institutional investors not only have reinforcement learning behavior in the decision-making, but also have game behaviors among institutional investors, namely, belief learning (game learning). EWA learning model [57] contains both reinforcement learning and belief learning. Therefore, we construct an evolution model of institutional investor behavior based on the EWA learning model:①Institutional investors reinforcement learning strategies: based on the EWA learning model,fit is defined as the attractiveness of strategy a to investor i at time t without considering the influence of other investors’ decisions, and fit∈−1,1 is satisfied. Suppose Qit represents the actual return if institutional investor i chooses strategy a at time t. λ is defined as the forgetting effect coefficient of investors. The larger the value of λ is, the less the influence of past experience is, and λ∈0,1 is satisfied. The rule for institutional investor to update the strategy based on reinforcement learning is(3)fiat=11+e−λe−λfiat−1+Qiat.②Institutional investment decision: supposeSit represents the strategy chosen by investor i at time t, Siat represents strategy a chosen by investor i at time t, and ISiat,Sit is the indicator function; when Siat=Sit, ISiat,Sit=1, and when Siat≠Sit, ISiat,Sit=0. S−it is a vector that represents the set of strategies adopted by all the other adjacent investors of investor i at time t. The return obtained by investor i choosing strategy a is πiSiat,S−it when the set of strategies adopted by other investors is S−it, πiSiat,S−it=ni−at−niat/nit. α is used to represent the intensity of returns competition among institutional investors; α∈0,1 is satisfied. Based on this, the decision function of institutional investor is(4)qiat=fiat+α·ISiat,Sit·πiSiat,S−it.According to the above analyses, the probability that institutional investori chooses to “buy” is pait=1=pfi1t>0,qi1t>0. The probability that institutional investors choose to “sell” is pait=−1=pfi−1t>0,qi−1t>0. The probability that institutional investors choose to “hold” is pait=−1=1=pait=1−pait=0.


## 2.1. Evolutionary Mechanism of Investor Behavior

Due to factors such as information asymmetry, the professionalism of investors, and differences in investment goals [44, 45], bounded rational investors are less likely to make optimal decisions only by observing the current stock market conditions. Experience is the easiest way to judge the feasibility of investment decisions. Before making the next investment decision, an investor usually refers to the returns obtained in the past. Even if the environment has changed, it will also repeat the strategy with better returns in the previous period [46, 47]. Therefore, the experience-based reinforcement learning method will directly affect investors’ investment strategies. However, there are significant differences between individual investors and institutional investors. Compared with individual investors, better performance can bring more capital inflows, which helps institutional investors achieve their goals, so there is intense performance competition among institutional investors [48]. Therefore, this paper divides investors into institutional investors and individual investors, the evolution model of investment behavior under different reinforcement learning strategies in network is constructed, as shown in Figure 1.Figure 1
The evolution mechanism of stock market investor’s decision.Based on the above analyses, the decision-making mechanism plays a crucial role in the evolution of investor behavior. In view of the differences between individual investors and institutional investors in investment decision-making, the learning strategies of investment behavior evolution of institutional investors and individual investors are constructed, respectively.


## 2.2. Evolutionary Model of Investor Behavior

SupposeG=N,E as the stock market, where N=1,2,…,i,…,n represents the investor and i,j∈E means the connection between investor i and investor j. ki refers to the degree of node i; pk represents degree distribution. Meanwhile, investors in the stock market are divided into institutional investors and individual investors, and the proportion of institutional investors in all investors is η0<η<1, so the proportion of individual investors is 1−η. In addition, it is assumed that there are three investment behavior states in stock market: ai∈−1,0,1, where ai=−1 means “sell,” ai=0 means “hold,” and ai=1 means “buy.” Based on this, the state of all investors in the stock market can be expressed as At=a1,t,a2,t,…,ai,t,…,aN,t at any time.mi0≤mi≤ki is set to indicate the number of individual investors which are associated with investor i, and ni0≤ni≤ki is set to indicate the number of institutional investors which are connected with investor i; then mi+ni=ki. Among them, it can be assumed that there are ni,1 buyers, ni,−1 sellers, and ni,0 holders of institutional investors associated with investor i, and ni,1+ni,−1+ni,0=ni. Similarly, it can be assumed that, among the individual investors associated with investor i, there are mi,1 buyers, mi,−1 sellers, and mi,0 holders, and mi,1+mi,−1+mi,0=mi.


### 2.2.1. Evolution Rules of Individual Investors Behavior

①Individual investors reinforcement learning strategies: Strahilevitz et al. [49] found that investors tend to repurchase the stocks they previously sold for a gain while ignoring the stocks they previously sold for a loss. Linnainmaa [50] found that household trading intensity depends on past performance. In the stock market, the investment performance of individual investors is largely affected by personal experience. In particular, investors tend to pay more attention to some successful experiences, even if such experience cannot be replicated in the future [51, 52]. These findings can be explained by reinforcement learning models. Individual investors will adjust their behavior according to the changes in the stock market at any time. On perceiving the changes of the market, they will take corresponding actions, such as buying, selling, or holding. The market will give certain feedback based on actions. If the action is successful, the probability of it being selected will increase [53]. Kaustia and Knüpfer [46] found that, compared with Bayesian learning theory, reinforcement learning theory can better explain the positive correlation between investors’ past IPO returns and future subscriptions. Choi et al. [54] used reinforcement learning theory to explain the savings behavior of individual investors. Erev and Roth [55] established a reinforcement learning model, which can well describe the evolution of individual behavior in economic experiments. Therefore, this paper refers to Erev and Roth’s reinforcement learning model to fit the evolution rules of investment behavior of individual investors.Supposeqi,at represents the attractiveness of strategy a to investor i at time t. Forgetting parameter λ is used to gradually weaken the influence of past experience, and λ∈0,1 always holds. ξit represents the return of investor i at time t. When the decision of investor i is the same as the optimal decision max∑kj∈Sikjaj,1,∑kj∈Sikjaj,−1,∑kj∈Sikjaj,0, his/her returns ξi=1. Otherwise, ξi=−1. The investor’s strategy update rules are as follows:(1)qi,at+1=11+e−λe−λqi,at+Ela,ξ,(2)Ela,ξ=ξ,a=l,0,a≠l.②Individual investment decisions: based on the above analyses, the probability that individual investori chooses to “buy” is pIait=1=qi,1t/∑aqi,at. The probability that individual investor i chooses to “sell” is pIait=−1=qi,−1t/∑aqi,at. The probability that individual investor chooses to “hold” is pIait=0=1−pait=1−pait=−1.


### 2.2.2. Evolution Rules of Institutional Investors Behavior

Greenwood and Nagel [56] analyzed the performance of experienced and inexperienced fund managers during the bubble. They found that inexperienced managers would buy technology stocks in large quantities during the run-up and sell them during the downturn. Hence, similar to individual investors, institutional investors also rely on reinforcement learning in trading. The historical performance of institutional investors will influence the capital flow and investment choice of individual investors. In order to maximize profits, there is a competitive relationship among institutional investors [48]. Compared with individual investors, institutional investors not only have reinforcement learning behavior in the decision-making, but also have game behaviors among institutional investors, namely, belief learning (game learning). EWA learning model [57] contains both reinforcement learning and belief learning. Therefore, we construct an evolution model of institutional investor behavior based on the EWA learning model:①Institutional investors reinforcement learning strategies: based on the EWA learning model,fit is defined as the attractiveness of strategy a to investor i at time t without considering the influence of other investors’ decisions, and fit∈−1,1 is satisfied. Suppose Qit represents the actual return if institutional investor i chooses strategy a at time t. λ is defined as the forgetting effect coefficient of investors. The larger the value of λ is, the less the influence of past experience is, and λ∈0,1 is satisfied. The rule for institutional investor to update the strategy based on reinforcement learning is(3)fiat=11+e−λe−λfiat−1+Qiat.②Institutional investment decision: supposeSit represents the strategy chosen by investor i at time t, Siat represents strategy a chosen by investor i at time t, and ISiat,Sit is the indicator function; when Siat=Sit, ISiat,Sit=1, and when Siat≠Sit, ISiat,Sit=0. S−it is a vector that represents the set of strategies adopted by all the other adjacent investors of investor i at time t. The return obtained by investor i choosing strategy a is πiSiat,S−it when the set of strategies adopted by other investors is S−it, πiSiat,S−it=ni−at−niat/nit. α is used to represent the intensity of returns competition among institutional investors; α∈0,1 is satisfied. Based on this, the decision function of institutional investor is(4)qiat=fiat+α·ISiat,Sit·πiSiat,S−it.According to the above analyses, the probability that institutional investori chooses to “buy” is pait=1=pfi1t>0,qi1t>0. The probability that institutional investors choose to “sell” is pait=−1=pfi−1t>0,qi−1t>0. The probability that institutional investors choose to “hold” is pait=−1=1=pait=1−pait=0.


## 2.2.1. Evolution Rules of Individual Investors Behavior

①Individual investors reinforcement learning strategies: Strahilevitz et al. [49] found that investors tend to repurchase the stocks they previously sold for a gain while ignoring the stocks they previously sold for a loss. Linnainmaa [50] found that household trading intensity depends on past performance. In the stock market, the investment performance of individual investors is largely affected by personal experience. In particular, investors tend to pay more attention to some successful experiences, even if such experience cannot be replicated in the future [51, 52]. These findings can be explained by reinforcement learning models. Individual investors will adjust their behavior according to the changes in the stock market at any time. On perceiving the changes of the market, they will take corresponding actions, such as buying, selling, or holding. The market will give certain feedback based on actions. If the action is successful, the probability of it being selected will increase [53]. Kaustia and Knüpfer [46] found that, compared with Bayesian learning theory, reinforcement learning theory can better explain the positive correlation between investors’ past IPO returns and future subscriptions. Choi et al. [54] used reinforcement learning theory to explain the savings behavior of individual investors. Erev and Roth [55] established a reinforcement learning model, which can well describe the evolution of individual behavior in economic experiments. Therefore, this paper refers to Erev and Roth’s reinforcement learning model to fit the evolution rules of investment behavior of individual investors.Supposeqi,at represents the attractiveness of strategy a to investor i at time t. Forgetting parameter λ is used to gradually weaken the influence of past experience, and λ∈0,1 always holds. ξit represents the return of investor i at time t. When the decision of investor i is the same as the optimal decision max∑kj∈Sikjaj,1,∑kj∈Sikjaj,−1,∑kj∈Sikjaj,0, his/her returns ξi=1. Otherwise, ξi=−1. The investor’s strategy update rules are as follows:(1)qi,at+1=11+e−λe−λqi,at+Ela,ξ,(2)Ela,ξ=ξ,a=l,0,a≠l.②Individual investment decisions: based on the above analyses, the probability that individual investori chooses to “buy” is pIait=1=qi,1t/∑aqi,at. The probability that individual investor i chooses to “sell” is pIait=−1=qi,−1t/∑aqi,at. The probability that individual investor chooses to “hold” is pIait=0=1−pait=1−pait=−1.


## 2.2.2. Evolution Rules of Institutional Investors Behavior

Greenwood and Nagel [56] analyzed the performance of experienced and inexperienced fund managers during the bubble. They found that inexperienced managers would buy technology stocks in large quantities during the run-up and sell them during the downturn. Hence, similar to individual investors, institutional investors also rely on reinforcement learning in trading. The historical performance of institutional investors will influence the capital flow and investment choice of individual investors. In order to maximize profits, there is a competitive relationship among institutional investors [48]. Compared with individual investors, institutional investors not only have reinforcement learning behavior in the decision-making, but also have game behaviors among institutional investors, namely, belief learning (game learning). EWA learning model [57] contains both reinforcement learning and belief learning. Therefore, we construct an evolution model of institutional investor behavior based on the EWA learning model:①Institutional investors reinforcement learning strategies: based on the EWA learning model,fit is defined as the attractiveness of strategy a to investor i at time t without considering the influence of other investors’ decisions, and fit∈−1,1 is satisfied. Suppose Qit represents the actual return if institutional investor i chooses strategy a at time t. λ is defined as the forgetting effect coefficient of investors. The larger the value of λ is, the less the influence of past experience is, and λ∈0,1 is satisfied. The rule for institutional investor to update the strategy based on reinforcement learning is(3)fiat=11+e−λe−λfiat−1+Qiat.②Institutional investment decision: supposeSit represents the strategy chosen by investor i at time t, Siat represents strategy a chosen by investor i at time t, and ISiat,Sit is the indicator function; when Siat=Sit, ISiat,Sit=1, and when Siat≠Sit, ISiat,Sit=0. S−it is a vector that represents the set of strategies adopted by all the other adjacent investors of investor i at time t. The return obtained by investor i choosing strategy a is πiSiat,S−it when the set of strategies adopted by other investors is S−it, πiSiat,S−it=ni−at−niat/nit. α is used to represent the intensity of returns competition among institutional investors; α∈0,1 is satisfied. Based on this, the decision function of institutional investor is(4)qiat=fiat+α·ISiat,Sit·πiSiat,S−it.According to the above analyses, the probability that institutional investori chooses to “buy” is pait=1=pfi1t>0,qi1t>0. The probability that institutional investors choose to “sell” is pait=−1=pfi−1t>0,qi−1t>0. The probability that institutional investors choose to “hold” is pait=−1=1=pait=1−pait=0.


## 3. Mathematical Analysis

Assuming that the degree of institutional investors isk at time t, the proportion of investment decision to buy is O1t, the proportion of investment decision to sell is O−1t, the proportion of the hold is O0t, and O1t+O−1t+O0t=η. Assuming that, at time t, the degree of individual investors is k, the proportion of investment decision to buy is I1t, the proportion of investment decision to sell is I−1t, and the proportion of choosing to hold is I0t, and I1t+I−1t+I0t=1−η.


### 3.1. Mathematical Analysis on the Evolution of Institutional Investor Behavior

Assume thatθO,1t represents the probability of connecting to institutional investors whose investment decision is buying; θO,1t=∑k≥1kpkO1t/k. Assume that θO,−1t represents the probability of connecting to institutional investors whose investment decision is selling; then θO,−1t=∑k≥1kpkO−1t/k. Assume that θO,0t represents the probability of connecting to institutional investors whose investment decision is holding; θO,0t=∑k≥1kpkO0t/k. Thus, the probability that there are exactly ni,1 buyers and ni,−1 sellers in the institutional investors connected to investor i is(5)ψt=kiniηni1−ηminini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1.So, the probability of institutional investori choosing to buy is(6)h1t=∑ni=0kikiniηni1−ηmi∑ni,1=0ni∑ni,−1=0ni−ni,1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1·pait=1.The probability of choosing to sell is(7)h−1t=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni−ni,−1nini,−1θO,−1tni,−1ni−ni,−1ni,1θO,−1tni,1θO,0tni−ni,1−ni,−1·pait=−1.The probability of choosing to hold is(8)h0t=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni−ni,−1nini,−1θO,−1tni,−1ni−ni,−1ni,1θO,−1tni,1θO,0tni−ni,1−ni,−1·pait=0.The rate of change that institutional investors buy stocks is(9)dO1tdt=O0t+O−1th1t−O1th0t+h−1t.According to the mean-field equation, the rate of change that institutional investors buy stocks is equal to the probability that the non-buyer converted into the buyer minus the probability of the buyer becoming a non-buyer. When the value of the above formula is equal to 0, the network reaches a steady state. We can infer(10)O1t=ηh1t.According to equation (10), combined with θO,1t=∑k≥1kpkO1t/k, the values of θO,1 and O1 in the equilibrium state of institutional investors behaviors can be determined:(11)θO,1=EθO,1=η∑k≥1kpkh1tk.However, it can be seen from the above analyses that it is difficult to see the impact of key elements such as network topology and learning strategies on the equilibrium state of the evolution of institutional investor behavior, so further discussion is needed.According topait=1=pfi1t>0,qi1t>0, it can be seen that investors will choose to buy when fi1t>αni,1−ni,−1/ni>0 and fi1t>0>αni,1−ni,−1/ni, so(12)h1t=∑ni=0kikiniηni1−ηmi∑ni,1=0ni∑ni,−1=0ni−ni,1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1·pait=1,=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni,−1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,+∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+nifi1t/αnini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1,θO,0tni−ni,1−ni,−1.In equation (12), ∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni,−1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1 is not affected by α, so it mainly considers the influence of α on ∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+nifi1t/αnini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1. The values of h1t do not change when α1>α2 and nifi1t/α1=nifi1t/α2, but the change is uncertain when α1>α2 and nifi1t/α1≠nifi1t/α2. Based on this, it is assumed that α1=nifi1t, α2=1/2nifi1t, and investors’ decision-making is not affected by α when ni=0 and fi1t<0, so analysis is not required. Therefore,(13)∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+nifi1t/α1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,−∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+nifi1t/α2nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,−∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+2nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,=−∑ni=0kikiniηni1−ηmi∑ni,−1=0ni−1∑ni,1=ni,−1+2ni,−1+2nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1<0,with the increase of α, h1t is not strictly decreasing; that is, EθO,1 is not strictly decreasing. However, it is difficult to obtain the influence of forgetting effect λ on the stable state θI,1 of institutional investor behavior through mathematical analysis, so this paper will analyze it by simulation.


### 3.2. Mathematical Analysis on the Evolution of Individual Investor Behavior

Assume thatθI,1t represents the probability of connecting to individual investors whose investment decision is buying; θI,1t=∑k≥1pkI1t/k. Assume that θI,−1t represents the probability of connecting to individual investors whose investment decision is selling; θI,−1t=∑k≥1kpkI−1t/k. Assume that θI,0t represents the probability of connecting to individual investors whose investment decision is holding; θI,0t=∑k≥1kpkI0t/k. Thus, the probability that there are exactly ni,1 buyers and ni,−1 sellers in the institutional investors connected to investor i is nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1. Similarly, the probability that there are exactly mi,1 buyers and mi,−1 sellers in the individual investors connected to investor i is mimi,1θI,1tmi,1mi−mi,1mi,−1θI,−1tmi,−1θI,0tmi−mi,1−mi,−1. The probability of the above two events occurring at the same time is(14)ϕ1t=kiniηni1−ηminini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1mimi,1θI,1tmi,1mi−mi,1mi,−1θI,−1tmi,−1θI,0tmi−mi,1−mi,−1.So, the probability of individual investori choosing to buy is(15)w1t=∑ni=0kikiniηni1−ηmi∑ni,1=0ni∑ni,−1=0ni−ni,1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1·∑mi,1=0mi∑mi,−1=0mi−mi,1mimi,1θI,1tmi,1mi−mi,1mi,−1θI,−1tmi,−1θI,0tmi−mi,1−mi,−1•pIait=1.The probability of choosing to sell is(16)w−1t=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni−ni,−1nini,−1θO,−1tni,−1ni−ni,−1ni,1θO,1tni,1θO,0tni−ni,1−ni,−1·∑mi,−1=0mi∑mi,1=0mi−mi,−1mimi,−1θI,−1tmi,−1mi−mi,−1mi,1θI,1tmi,1θI,0tmi−mi,1−mi,−1·pIait=−1.The probability of choosing to hold is(17)w0t=∑ni=0kikiniηni1−ηmi∑ni,1=0ni∑ni,−1=0ni−ni,1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1·∑mi,1=0mi∑mi,−1=0mi−mi,1mimi,1θI,1tmi,1mi−mi,1mi,−1θI,−1tmi,−1θI,0tmi−mi,1−mi,−1·pIait=0.The rate of change that individual investors buy stocks is(18)dI1tdt=I0t+I−1tw1t−I1tw0t+w−1t.According to the mean-field equation, the rate of change that individual investors buy stocks is equal to the probability that the non-buyer converted into the buyer minus the probability of the buyer becoming a non-buyer. When the value of the above formula is equal to 0, the network reaches a steady state. We can infer(19)I1t=1−ηw1t.According to equation (19), combined with θI,1=∑k≥1kpkI1t/k, the values of θI,1t and I1 in the equilibrium state of individual investors behaviors can be determined:(20)θI,1=1−η1k∑k≥1kpkw1t.However, it can be seen that the influence of learning strategy, forgetting effect, initial state, and other factors on the equilibrium state of individual investor behavior cannot be obtained through mathematical analysis. Therefore, this paper will analyze it by simulation.


## 3.1. Mathematical Analysis on the Evolution of Institutional Investor Behavior

Assume thatθO,1t represents the probability of connecting to institutional investors whose investment decision is buying; θO,1t=∑k≥1kpkO1t/k. Assume that θO,−1t represents the probability of connecting to institutional investors whose investment decision is selling; then θO,−1t=∑k≥1kpkO−1t/k. Assume that θO,0t represents the probability of connecting to institutional investors whose investment decision is holding; θO,0t=∑k≥1kpkO0t/k. Thus, the probability that there are exactly ni,1 buyers and ni,−1 sellers in the institutional investors connected to investor i is(5)ψt=kiniηni1−ηminini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1.So, the probability of institutional investori choosing to buy is(6)h1t=∑ni=0kikiniηni1−ηmi∑ni,1=0ni∑ni,−1=0ni−ni,1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1·pait=1.The probability of choosing to sell is(7)h−1t=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni−ni,−1nini,−1θO,−1tni,−1ni−ni,−1ni,1θO,−1tni,1θO,0tni−ni,1−ni,−1·pait=−1.The probability of choosing to hold is(8)h0t=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni−ni,−1nini,−1θO,−1tni,−1ni−ni,−1ni,1θO,−1tni,1θO,0tni−ni,1−ni,−1·pait=0.The rate of change that institutional investors buy stocks is(9)dO1tdt=O0t+O−1th1t−O1th0t+h−1t.According to the mean-field equation, the rate of change that institutional investors buy stocks is equal to the probability that the non-buyer converted into the buyer minus the probability of the buyer becoming a non-buyer. When the value of the above formula is equal to 0, the network reaches a steady state. We can infer(10)O1t=ηh1t.According to equation (10), combined with θO,1t=∑k≥1kpkO1t/k, the values of θO,1 and O1 in the equilibrium state of institutional investors behaviors can be determined:(11)θO,1=EθO,1=η∑k≥1kpkh1tk.However, it can be seen from the above analyses that it is difficult to see the impact of key elements such as network topology and learning strategies on the equilibrium state of the evolution of institutional investor behavior, so further discussion is needed.According topait=1=pfi1t>0,qi1t>0, it can be seen that investors will choose to buy when fi1t>αni,1−ni,−1/ni>0 and fi1t>0>αni,1−ni,−1/ni, so(12)h1t=∑ni=0kikiniηni1−ηmi∑ni,1=0ni∑ni,−1=0ni−ni,1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1·pait=1,=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni,−1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,+∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+nifi1t/αnini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1,θO,0tni−ni,1−ni,−1.In equation (12), ∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni,−1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1 is not affected by α, so it mainly considers the influence of α on ∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+nifi1t/αnini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1. The values of h1t do not change when α1>α2 and nifi1t/α1=nifi1t/α2, but the change is uncertain when α1>α2 and nifi1t/α1≠nifi1t/α2. Based on this, it is assumed that α1=nifi1t, α2=1/2nifi1t, and investors’ decision-making is not affected by α when ni=0 and fi1t<0, so analysis is not required. Therefore,(13)∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+nifi1t/α1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,−∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+nifi1t/α2nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,−∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=ni,−1+1ni,−1+2nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1,=−∑ni=0kikiniηni1−ηmi∑ni,−1=0ni−1∑ni,1=ni,−1+2ni,−1+2nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1<0,with the increase of α, h1t is not strictly decreasing; that is, EθO,1 is not strictly decreasing. However, it is difficult to obtain the influence of forgetting effect λ on the stable state θI,1 of institutional investor behavior through mathematical analysis, so this paper will analyze it by simulation.


## 3.2. Mathematical Analysis on the Evolution of Individual Investor Behavior

Assume thatθI,1t represents the probability of connecting to individual investors whose investment decision is buying; θI,1t=∑k≥1pkI1t/k. Assume that θI,−1t represents the probability of connecting to individual investors whose investment decision is selling; θI,−1t=∑k≥1kpkI−1t/k. Assume that θI,0t represents the probability of connecting to individual investors whose investment decision is holding; θI,0t=∑k≥1kpkI0t/k. Thus, the probability that there are exactly ni,1 buyers and ni,−1 sellers in the institutional investors connected to investor i is nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1. Similarly, the probability that there are exactly mi,1 buyers and mi,−1 sellers in the individual investors connected to investor i is mimi,1θI,1tmi,1mi−mi,1mi,−1θI,−1tmi,−1θI,0tmi−mi,1−mi,−1. The probability of the above two events occurring at the same time is(14)ϕ1t=kiniηni1−ηminini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1mimi,1θI,1tmi,1mi−mi,1mi,−1θI,−1tmi,−1θI,0tmi−mi,1−mi,−1.So, the probability of individual investori choosing to buy is(15)w1t=∑ni=0kikiniηni1−ηmi∑ni,1=0ni∑ni,−1=0ni−ni,1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1·∑mi,1=0mi∑mi,−1=0mi−mi,1mimi,1θI,1tmi,1mi−mi,1mi,−1θI,−1tmi,−1θI,0tmi−mi,1−mi,−1•pIait=1.The probability of choosing to sell is(16)w−1t=∑ni=0kikiniηni1−ηmi∑ni,−1=0ni∑ni,1=0ni−ni,−1nini,−1θO,−1tni,−1ni−ni,−1ni,1θO,1tni,1θO,0tni−ni,1−ni,−1·∑mi,−1=0mi∑mi,1=0mi−mi,−1mimi,−1θI,−1tmi,−1mi−mi,−1mi,1θI,1tmi,1θI,0tmi−mi,1−mi,−1·pIait=−1.The probability of choosing to hold is(17)w0t=∑ni=0kikiniηni1−ηmi∑ni,1=0ni∑ni,−1=0ni−ni,1nini,1θO,1tni,1ni−ni,1ni,−1θO,−1tni,−1θO,0tni−ni,1−ni,−1·∑mi,1=0mi∑mi,−1=0mi−mi,1mimi,1θI,1tmi,1mi−mi,1mi,−1θI,−1tmi,−1θI,0tmi−mi,1−mi,−1·pIait=0.The rate of change that individual investors buy stocks is(18)dI1tdt=I0t+I−1tw1t−I1tw0t+w−1t.According to the mean-field equation, the rate of change that individual investors buy stocks is equal to the probability that the non-buyer converted into the buyer minus the probability of the buyer becoming a non-buyer. When the value of the above formula is equal to 0, the network reaches a steady state. We can infer(19)I1t=1−ηw1t.According to equation (19), combined with θI,1=∑k≥1kpkI1t/k, the values of θI,1t and I1 in the equilibrium state of individual investors behaviors can be determined:(20)θI,1=1−η1k∑k≥1kpkw1t.However, it can be seen that the influence of learning strategy, forgetting effect, initial state, and other factors on the equilibrium state of individual investor behavior cannot be obtained through mathematical analysis. Therefore, this paper will analyze it by simulation.


## 4. Simulation Results

The previous part has theoretically analyzed the evolution mechanism of investor behavior in the stock market under the reinforcement learning. It can be seen from the analysis that the evolution of the stock market investor behavior on the reinforcement learning is mainly affected by factors such as the network structure, the intensity of returns competition among institutional investorsα, and forgetting parameters λ. On this basis, this section mainly employs simulation, to further explain the evolution process of investor behavior. According to the statistical yearbook data of the Shanghai Stock Exchange, the net sales of various types of investors in the past five years are as follows: the proportion of natural person investor transactions is 80% to 85%, the proportion of transactions of general legal persons, Shanghai Stock Connect, and professional institutions ranges from 15% to 20%. Based on this, this paper sets the proportion of institutional investors in the network to η=0.2. If the total number of investors is set to 500, the number of individual investors is 400 and the number of institutional investors is 100. Existing research shows that the network of stock investors based on financial correlation exhibits power-law distribution characteristics [18–20]. Based on this, a two-layer scale-free network is constructed, and the average degree of network is different at different layers. The edges of different types of investors are established according to degree-dependent connection and random connection, respectively, in this paper. In the initial state, all investors are in the “holding” state. The evolution time is 300. The default values of all parameters are λ=0.5, α=0.1, Qit∈−1,1, and the average degree of the entire network is <k>=12. This paper assumes that the market is generally in a good state.


### 4.1. Impact of Network Structure on the Evolution of Investor Behavior in Stock Market

Figure2 compares and analyzes the dynamic evolution of the two types of investor behavior in two network structures of random connection and degree-dependent connection when the network average degree is <k>=8 and <k>=12, respectively.Figure 2
The impact of network structure on the evolution of investor behavior.
(a)(b)(c)(d)Figure2 describes the impact of network connectivity and network heterogeneity on the evolution of investor behavior. For individual investors, Figures 2(a) and 2(b) show that, in BA-BA randomly connected multi-layer networks, the time for the individual investor’s behavior evolution to reach a steady state is t≈40 when <k>=8 and the time is t≈10 when <k>=12. It indicates that the time for investment behavior to evolve into steady state shortens as the average degree increases. Figures 2(c) and 2(d) show that, in a BA-BA degree related connection multi-layer network, the time for the evolution of individual investor behavior to reach steady state is t≈70 when <k>=8 and the time is t≈10 when <k>=12.This shows that network connectivity and network heterogeneity have an impact on the evolution of individual investor behavior.For institutional investors, in any case, the dynamic evolution of their behavior quickly reaches a steady state, and the realization time and the state ratio are basically the same. It can be seen that the network topology significantly affects the behavioral evolution of individual investors, but it has a weaker impact on institutional investors.


### 4.2. Impact of the Intensity of Returns Competition among Institutional Investors on the Evolution of Stock Market Investor Behavior

In the case where all the parameter values are the same, Figure3 analyzes the evolution of the investment behavior of institutional investors and individual investors under different intensity of returns competition among institutional investors α.Figure 3
The impact ofα on the evolution of investor behavior.
(a)(b)As shown in Figure3, under the two network structures, for institutional and individual investors, the intensity of returns competition among institutional investors α has an impact on their investment behavior. For institutional investors, with the increase of α, the proportion of institutional investors “buying” has continued to decrease, and the proportion of “holding” has continued to increase, while the proportion of “selling” has not changed significantly. Compared with institutional investors, with the increase of α, the proportion of individual investors “holding” and “selling” has increased slightly, and the proportion of “buying” has continued to decrease.


### 4.3. Impact of Forgetting Effect on the Evolution of Investor Behavior in Stock Market

This section mainly analyzes the impact of forgetting effect coefficient on the dynamic evolution of investor behavior in two network structures, respectively. Other experimental parameters are set with the above default values. The simulation results are shown in Figure4.Figure 4
The impact ofλ on the evolution of investor behavior.
(a)(b)As shown in Figure4, under two network structures, with the increase of the forgetting coefficient λ, the proportion of individual investors “buying” is continuously decreasing, and the proportion of “holding” and “selling” increases slightly. Compared with the inter-layer random connection network, the proportion of individual investors “buying” is higher in the inter-layer degree-dependent connection network. Unlike the behavior of individual investors that is obviously affected by the forgetting effect, institutional investors are relatively less affected by the forgetting coefficient. As the forgetting coefficient λ increases, the proportion of institutional investors “buying” decreases to a lesser extent. In summary, with the increase of the investor’s forgetting effect coefficient, the proportion of investors who “buy” at equilibrium is reduced; compared with institutional investors, the proportion of “buyers” of individual investors decreases even more. That is, the forgetting effect has a greater impact on individual investor behavior.


### 4.4. Impact of Initial Investment Behavior State on the Evolution of Investor Behavior in Stock Market

This section mainly discusses the effect of the initial investment behavior of investors on the dynamic evolution process of their behaviors. The parameters in the simulation refer to the above default values.Figures5(a)–5(d) describe the effect of the initial behavioral status of investors on the dynamic evolution of their behaviors in the two types of network structures with average degrees of k=8 and k=12, respectively. It can be seen from the figure that as the proportion of investors’ initial investment behaviors “buy” status gradually increases, the ultimate equilibrium state of institutional investor and individual investor investment behavior evolution are basically the same. It shows that, whether it is an institutional investor or an individual investor, the initial investment behavior status has little effect on its behavior evolution.Figure 5
The impact of the initial state on the evolution of investor behavior.
(a)(b)(c)(d)


## 4.1. Impact of Network Structure on the Evolution of Investor Behavior in Stock Market

Figure2 compares and analyzes the dynamic evolution of the two types of investor behavior in two network structures of random connection and degree-dependent connection when the network average degree is <k>=8 and <k>=12, respectively.Figure 2
The impact of network structure on the evolution of investor behavior.
(a)(b)(c)(d)Figure2 describes the impact of network connectivity and network heterogeneity on the evolution of investor behavior. For individual investors, Figures 2(a) and 2(b) show that, in BA-BA randomly connected multi-layer networks, the time for the individual investor’s behavior evolution to reach a steady state is t≈40 when <k>=8 and the time is t≈10 when <k>=12. It indicates that the time for investment behavior to evolve into steady state shortens as the average degree increases. Figures 2(c) and 2(d) show that, in a BA-BA degree related connection multi-layer network, the time for the evolution of individual investor behavior to reach steady state is t≈70 when <k>=8 and the time is t≈10 when <k>=12.This shows that network connectivity and network heterogeneity have an impact on the evolution of individual investor behavior.For institutional investors, in any case, the dynamic evolution of their behavior quickly reaches a steady state, and the realization time and the state ratio are basically the same. It can be seen that the network topology significantly affects the behavioral evolution of individual investors, but it has a weaker impact on institutional investors.


## 4.2. Impact of the Intensity of Returns Competition among Institutional Investors on the Evolution of Stock Market Investor Behavior

In the case where all the parameter values are the same, Figure3 analyzes the evolution of the investment behavior of institutional investors and individual investors under different intensity of returns competition among institutional investors α.Figure 3
The impact ofα on the evolution of investor behavior.
(a)(b)As shown in Figure3, under the two network structures, for institutional and individual investors, the intensity of returns competition among institutional investors α has an impact on their investment behavior. For institutional investors, with the increase of α, the proportion of institutional investors “buying” has continued to decrease, and the proportion of “holding” has continued to increase, while the proportion of “selling” has not changed significantly. Compared with institutional investors, with the increase of α, the proportion of individual investors “holding” and “selling” has increased slightly, and the proportion of “buying” has continued to decrease.


## 4.3. Impact of Forgetting Effect on the Evolution of Investor Behavior in Stock Market

This section mainly analyzes the impact of forgetting effect coefficient on the dynamic evolution of investor behavior in two network structures, respectively. Other experimental parameters are set with the above default values. The simulation results are shown in Figure4.Figure 4
The impact ofλ on the evolution of investor behavior.
(a)(b)As shown in Figure4, under two network structures, with the increase of the forgetting coefficient λ, the proportion of individual investors “buying” is continuously decreasing, and the proportion of “holding” and “selling” increases slightly. Compared with the inter-layer random connection network, the proportion of individual investors “buying” is higher in the inter-layer degree-dependent connection network. Unlike the behavior of individual investors that is obviously affected by the forgetting effect, institutional investors are relatively less affected by the forgetting coefficient. As the forgetting coefficient λ increases, the proportion of institutional investors “buying” decreases to a lesser extent. In summary, with the increase of the investor’s forgetting effect coefficient, the proportion of investors who “buy” at equilibrium is reduced; compared with institutional investors, the proportion of “buyers” of individual investors decreases even more. That is, the forgetting effect has a greater impact on individual investor behavior.


## 4.4. Impact of Initial Investment Behavior State on the Evolution of Investor Behavior in Stock Market

This section mainly discusses the effect of the initial investment behavior of investors on the dynamic evolution process of their behaviors. The parameters in the simulation refer to the above default values.Figures5(a)–5(d) describe the effect of the initial behavioral status of investors on the dynamic evolution of their behaviors in the two types of network structures with average degrees of k=8 and k=12, respectively. It can be seen from the figure that as the proportion of investors’ initial investment behaviors “buy” status gradually increases, the ultimate equilibrium state of institutional investor and individual investor investment behavior evolution are basically the same. It shows that, whether it is an institutional investor or an individual investor, the initial investment behavior status has little effect on its behavior evolution.Figure 5
The impact of the initial state on the evolution of investor behavior.
(a)(b)(c)(d)


## 5. Conclusion

In this paper, we analyze the decision-making mechanism of individual investors and institutional investors on the theories of complex networks and reinforcement learning and construct the evolution model of investors behavior based on the reinforcement learning in network. Then, the evolution characteristics of investors behavior are investigated by mean-field analysis and simulations. The conclusions are drawn as follows: (1) the topology of the network has a greater influence on the evolution of the individual investor behavior than that of the institutional investor. (2) The equilibrium state of institutional investors behavior is in nonlinear correlation with the intensity of returns competition among institutional investors. Moderate competition for returns among institutional investors is conducive to the stability of the stock market. (3) As to all institutional investors and individual investors, the forgetting effect has a significant influence on the equilibrium of their evolution. Through reinforcement learning, investors can obtain useful investment advice from a large amount of market information, choose which information is more worthy of attention, and to a certain extent eliminate the influence of behavioral bias on investment, but at the same time, the successful experiences will also strengthen the irrational characteristics of investors, such as overconfidence, etc. In a bull market, investors often attribute their success to their investment ability, which makes them more overconfident and further leads to irrational behaviors. Therefore, investors should view their own experience as rationally as possible when making investment decisions and establish the belief of long-term investment. (4) The initial state of the investors behavior has a small influence on the equilibrium of their evolution for all institutional investors and individual investors.This paper mainly studies the evolution of investor behavior based on the static network structure. However, the network structure is bound to evolve with the change of investment subject’s behavior. Therefore, it will be the authors’ future work to analyze the evolution of investor behavior under dynamic network structure.



---

*Source: 3561538-2020-09-28.xml*