# Algorithmic Forex Trading Using Q-learning

Hasna Haifa Zahrah, Jimmy Tirtawangsa

► To cite this version:

Hasna Haifa Zahrah, Jimmy Tirtawangsa. Algorithmic Forex Trading Using Q-learning. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.24-35, 10.1007/978-3-031-34111-3_3. hal-05449796

HAL Id: hal-05449796

https://inria.hal.science/hal-05449796v1

Submitted on 9 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Algorithmic Forex Trading Using Q-learning

Hasna Haifa Zahrah(✉) and Jimmy Tirtawangsa

Telkom University, Bandung, Indonesia

hanazahrah30@gmail.com, jimmytirtawangsa@telkomuniversity.ac.id

**Abstract.** The forex market is a difficult market for traders to succeed. The high noise and volatility of the forex market make the traders very hard to open and close position accurately. Many approaches have been proposed to overcome these difficulties, including algorithmic trading. This research proposed a framework for algorithmic trading using Q-learning with the help of LSTM. The proposed framework uses a finite state space in reinforcement learning to use holding time and higher timeframe market data. The state space is designed so that the agent can open and close positions flexibly, without being restricted by a fixed time window. This allows the agent to take profits and avoid losses. The proposed framework was trained and tested using 15 years' worth of historical data of the EUR/USD currency pair in 5-min timeframe data. The system was evaluated based on various metrics such as profit, drawdown, Sharpe ratio, holding time, and delta time. The results show that with its designed finite state space and flexible time window, the proposed framework achieved consistent profits, reduced losses, and increased overall profits. This suggests that the proposed framework may be a suitable solution for forex market trading.

**Keywords:** Q-learning · LSTM · Forex

## 1 Background

The foreign exchange (forex) market is a financial market where currency trading takes place. Participants in this market range from large commercial banks to small retail traders who require a broker to access the trading platform. As a result, the forex market has become popular among retail speculators or day traders. Despite its numerous participants, Hayley and Marsh [1] reported a high rate of Forex traders losing money. This failure rate is largely due to various market characteristics [2], such as the influence of economics, politics, society, and traders' psychology.

These factors often cause traders to act impulsively, leading to noise in price data and distorted genuine market trends. The noise can also come from broker manipulation of market prices and volatility, which makes it difficult to determine when to open or close a position. These actions can significantly impact a trader's losses and profits in forex trading. The forex market has various time frame markets with its own level of volatility. The higher time frame market provides a larger picture of the price movement, while the
lower time frame market provides detailed information. There are correlations between each time frame market that can be useful to forecast price trend of forex market.

Shavandi et al. [3] proposed a multi-agent deep reinforcement learning framework to trade in the forex market by utilizing multiple time frame markets, resulting in a 22.7% higher average cumulative return compared to a single time frame method. The research uses a fixed time window to open and close positions, aiming for consistent profits as reflected in the Sharpe ratio evaluation of 0.63. However, this fixed time window may miss potential profits if the position is closed before desired profit is achieved. Furthermore, the position could not close the position flexibly resulting in avoidable losses that may not occur if the position close before the loss.

Therefore, Removing the fixed time window can be a challenging task as it could result in inconsistent profits with potentially high loss values. However, the fixed time window may also lead to missed profits and unnecessary losses that could have been avoided. The question arises: Will removing the time window lead to increased profits and decreased loss because the time window does not restrict it?

This research aims to create a framework that facilitates the flexible opening and closing of positions in the forex market to achieve more profitable results. The framework employs reinforcement learning with LSTM to predict buying, selling, or holding actions based on price trends as the starting prediction rather than relying on random numbers. The use of Reinforcement Learning in the framework seeks to regulate losses and achieve profitable results by re-designing the state and Q-function. Prior studies on the finite state were conducted on the stock market [4], not the forex market, as most research [5-7] on the forex market utilized deep reinforcement learning. However, the previous research [4] only considered the historical price, whereas this study also considers other variables such as holding time to determine the optimal length of holding a trade position in addressing the research question.

The hypothesis for the research question is removing the fixed time window might lead to increased profit and decreased loss because there is no set time limit to open and close the position. Therefore, the independent variable is the time window of holding position and not opening a position in the forex market, while the dependent variables are profit and drawdown.

# 2 Related Studies

## 2.1 Fundamental Analysis

There are two main methods for trading in the forex market: fundamental analysis and technical analysis. Fundamental analysis considers economic, political, and social events and news that could have an impact on the forex market. An example of trading using fundamental analysis would be taking advantage of the release of non-farm payrolls news. This news reports the number of workers in the US, excluding those in agriculture and non-profit organizations. If the actual number is different from the expected number, it can cause a change in the price trend and provide an opportunity for a trader to open a position.
## 2.2 Technical Analysis

The other method is technical analysis, which is based solely on historical market data that reflect market participants' behavior towards economic, political, and social events [8]. Technical analysis employs several techniques, including technical indicators that are extracted to summarize market conditions, which can then be manually implemented by humans. These indicators may be based on mathematical calculations to predict price trends, such as moving average and relative strength indicator. Others may be based on market chart patterns, such as head shoulder or Elliot wave.

## 2.3 Algorithmic Trading

Another growing technique, particularly in the research field, is algorithmic trading. This is a method of trading that uses algorithms to automatically place and execute orders in the market [9]. These algorithms may contain simple rules, such as taking profit and stopping loss orders based on the current price and a certain length of time, or more complex methods, such as machine learning.

An example of using fundamental analysis for algorithmic trading can be found in [10]. This study utilized fundamental analysis to validate decisions made by other system strategies. Another research [11] proposed an expert advisor that leverages macroeconomic events to detect patterns and offer trading signals to day traders in the forex market. Another study [12] applied algorithmic trading using technical indicators such as RSI and MACD and achieved good results with returns up to 12%.

Reinforcement learning (RL), which is a type of machine learning, is a method in which a system learns in a certain environment by taking actions based on the given state and receiving rewards as feedback to determine if the action is good or bad. This method can replicate trading in the forex market, where the environment is the market with certain trading strategy rules. The method has also been implemented in the financial field, specifically in the forex market. In 2018 [5], deep RL was applied to forex trading algorithms with action augmentation techniques and produced positive returns in 12 different currency pairs. The same year, a forex trading system was proposed using RL with Q-networks [6], which produced an average total profit of 114.0 ± 19.6%. In 2019 [7] proposed a RL method with the addition of trailing rewards for forex trading and obtained a profit of 28.6%.

# 3 Data

## 3.1 Data Description

The proposed system takes as input historical EUR/USD 5-min timeframe data similar to the baseline research [3], which was downloaded from Alpari Securities through the MetaTrader5 platform. Additionally, higher timeframe data such as 1-h and 4-h is also utilized to implement multiple time frame markets. However, this research trades in 5-min timeframe data. The data consists of date, timestamp, high price, open price, close price, low price, and volume. However, the volume feature is not utilized in this research. The data were collected from 2008 to 2016 and were used for training, which resulted in
a total of 668, 752 rows. Meanwhile, 446, 458 rows of data from 2017 to 2022 were used as testing data. It's worth noting that the baseline research[3] used different historical data, specifically from 06/29/2012 to 05/25/2021, but still comparable results can be obtained as this research and baseline research used multiple years of historical data.

## 3.2 Data Preprocessing

This study had three possible bar trends to occur: an uptrend, a downtrend, and a flat trend which was also referred to as a no trend. To determine the trend, the close price and open price of each row data were compared. If the close price was higher than the open price, the trend was considered an uptrend. On the other hand, if the close price was lower than the open price, the trend was considered a downtrend.

In situations where the difference between the closing price and opening price was less than 3 pips, the trend was considered a flat trend. A pip is a unit of measurement used in trading and refers to the fourth decimal place in a currency pair's price. The threshold for determining the bar data as a flat varies for each time frame. 3 pips for 5-min, 5 pips for 1-h, and 10 pips for 4-h.

The action, profit, and loss values were calculated within a window data consisting of 48 rows of 5-min data, which were equivalent to 4-h. The range of window data was determined by the 4-h duration of the overlapping London and New York market sessions, which had the highest trading volume for EUR/USD and resulted in significant price movements. The window data shifted every one row; hence the current row data were the 48th data of the current window data and the next row data would be the 48th data of the next window data.

There are three values of action: buy, sell, and hold. The action value is determined by comparing the first close price and the last close price within the window data. If the last close price is higher than the first close price, the action is assigned as 'buy' to indicate opening a long position. Conversely, if the last close price is lower than the first close price, the action is assigned as 'sell' to indicate opening a short position. Long position is when a trader expects the price to move upward, while a short position is when they expect it to move downward. If the first and last close prices are the same, the action is assigned as 'hold'.

The profit was a positive number calculated as the earnings from opening and closing a position. The position was opened at the first data and closed at the last data within the window data. Therefore, the difference price between the last and the first data within the window data were calculated to get profit. The loss within the range of opening and closing a position was also calculated. Furthermore, if the value of loss was bigger than the profit value then the action was changed to 'hold' indicating a decision not to participate in the market as the chosen action results in a loss. The label for each row data was based on the action value in the next row data.

## 4 LSTM Training

The LSTM input was a window data consisting of 48 data with three features: bar trend 5-min, action, and profit loss. The training of LSTM model uses training data as described in Sect. 3.1 with 20 epochs. At each time step *t* the window data contained data from
row $t-47$ to $t$, hence the time step value started at 48. The label for the window data is the label value from row data $t$ as described in Sect. 3.2. The window data is shifting by one step at each time step. The use of window data enables the LSTM model to identify patterns in the data and predict the next data trend.

The architecture of the LSTM model consists of a single layer of 64 units with a tanh activation function. In the output layer, a SoftMax activation is employed to generate the probability for each of the labels: buy, hold, and sell. The predicted action is chosen by selecting the label with the highest probability. Adam optimization algorithm is used to train the LSTM model with default hyperparameters. The output of the LSTM training is LSTM model with the best weight utilized in Reinforcement Learning training process.

# 5 Reinforcement Learning

The main model utilized in this research is the Reinforcement Learning model, which is composed of an Environment and an Agent as illustrated in Fig. 1. During training, the Environment received a predicted action from a previously trained LSTM. On the other hand, during testing, the Environment received a chosen action by the Agent using the Q-table trained with Q-learning algorithm.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Proposed Reinforcement Learning model</figcaption></figure>

## 5.1 Reward Calculation in Environment

The environment outputs a reward to indicate whether the action taken by the agent was good or bad. The profit (P) was calculated as the difference between the close price at the current time step and the close price at the time the position was opened, as shown in Eq. 1.

$$P_t = (cp_t - cp_o) \times 10000 \quad (1)$$

where $t$ is a time step, $cp_t$ is a close price of time step $t$ and $cp_o$ is a close price of open position. The result was multiplied by 10,000 to express it in pips, which is a tradeable unit as described in Sect. 3.2. Positive rewards were expressed as profit, while negative rewards were expressed as drawdown. The drawdown (d) as shown in Eq. 2
was calculated as the difference between the lowest close price before the agent closes the position and the close price at the open position.

$$d = (cp_{\text{lowest\_price}} - cp_{\text{o}}) \times 10000 \quad (2)$$

Afterwards, the reward was calculated using profit, drawdown, and holding time as shown in Eq. 3. The holding time was the total number of bar data which the open position has been held.

$$R_t = \frac{\left( \frac{P_t}{\text{holding\_time}} \right) - (d \times \text{holding\_time})}{(cp_o \times 10000)} \quad (3)$$

The purpose of dividing the profit by holding time is to reduce the profit as the length of time a position is held increases. On the other hand, multiplying the drawdown by holding time maximizes the loss, meaning that holding a position for too long results in a small positive reward and a large negative reward. This calculation was used to teach the agent to not keep a position open for an excessive amount of time.

## 5.2 State Construction in Environment

This process generates a state that represents the information available when selecting an action. The state comprises various components as provided in Table 1.

<table><caption>**Table 1.** State components</caption><thead><tr><th>Component</th><th>Value</th><th>Purpose</th></tr></thead><tbody><tr><td>Position</td><td>long, no, shortposition</td><td>To define the position of agent based on action</td></tr><tr><td>Bar trend 5-min</td><td>uptrend, flat, downtrend</td><td>To know what the current trend of 5-min price is</td></tr><tr><td>Reward</td><td>+, 0, -</td><td>To Indicate whether the selected action is good or bad</td></tr><tr><td>Holding time or Delta time</td><td>30 min, 1 h, 4 h, one day</td><td>To Calculate the number of bar data that the open position has been held or the number of bar data that the agent does not open a position</td></tr><tr><td>Target pips</td><td>3, 10, 17, 27</td><td>To learn the closing position based on the obtained profit</td></tr><tr><td>Bar trend of higher timeframe</td><td>bar trend of 5-min = bar trend of 1-h or 4-h timeframe</td><td>To know whether the current 5-min trend matches the current higher time frame trend</td></tr></tbody></table>
## 5.3 Agent

The Agent was expected to select action from the Q-table. This was achieved by identifying the action with the highest Q-value for the current state provided by the Environment. The Q-table was a $n \times 3$ matrix where $n$ represented the total number of determined states, and 3 represented the total number of actions, ‘buy’, ‘sell’, and ‘hold’. At the beginning, the Q-table was empty, containing only zeros. Subsequently, during each iteration, the Q-function was used to update the values of each state-action pair in the Q-table. Temporal difference formula as shown in Eq. 4 was used to update the Q-value of the current state and action at time step $t$ using the obtained reward and Q-value of the next state.

$$Q^{\pi}(s_t, a_t) = (1 - \alpha)Q^{\pi}(s_t, a_t) + \frac{\alpha(R_t + \gamma Max(Q^{\pi}(s_{t+1}))))}{2} \quad (4)$$

where $s$, $a$, $t$, $R$, $\alpha$, and $\gamma$ are state, action, time step, reward, learning rate, and discounted factor, respectively. $\alpha$ was set with values of 0.85 to give a fair amount of update to current Q-value. Meanwhile, a discount factor of 0.99 was set as $\gamma$ to indicate that the total expected future reward was discounted by a small amount, but still considered in the calculation.

The division by 2 is the modification of temporal difference formula proposed in this research. It serves to update the Q-value with the total reward of the current state, rather than the expected future reward, so that the Q-value does not continue to increase indefinitely. Equation 4 aims to give a higher Q-value by selecting the maximum value of the next Q-value. It indicates that selected action $a$ for state $s$ obtained a positive reward.

On the other hand, for negative rewards, which are present in this research, the selection of maximum next Q-value is changed to the minimum as shown in Eq. 5. This modification is made to assign a lower Q-value to the selected action $a$ for the state $s$ that results in negative reward, thus preventing the agent from selecting action $a$ when receiving state $s$ from the environment.

$$Q^{\pi}(s_t, a_t) = (1 - \alpha)Q^{\pi}(s_t, a_t) + \frac{\alpha(R_t + \gamma Min(Q^{\pi}(s_{t+1}))))}{2} \quad (5)$$

## 5.4 Reinforcement Learning Training

The aim of the Reinforcement Learning training process is to update the Q-table, which is utilized by the Agent to select the best action when receiving a state from the Environment. In this training process, a random action is selected based on the state given by the Environment. The reward and next state obtained from this action are then used to update the Q-value for the current action and state using the Q-function as described in Sect. 5.3.

However, in this research, rather than selecting a random action when receiving a state from the Environment, a predicted action from a trained LSTM is used, as shown in Fig. 1. In each iteration, the Environment provides the LSTM model with a bar trend of 5-min data, position, and reward. The LSTM model then predicts the action. Additionally,
a state that includes the bar trend of 5-min data, position, and reward is output from the Environment, along with the reward, and is given to the Agent to update the Q-value of the predicted action and state in the Q-table. After all the iterations, the updated Q-table is then utilized in the testing process.

## 5.5 Reinforcement Learning Testing and Evaluation

The Q-table generated from the training process is utilized by the agent to make decisions based on the state input received from the environment. It is expected that the Q-table contains optimized information from the LSTM, thus eliminating the need for the LSTM model to determine actions. At every time step, the environment gives the agent a state, and the agent selects the best action based on the received state by using the Q-table, as depicted in Fig. 1. After a predetermined number of timesteps, the agent will have the final profit calculated from the testing simulation.

The evaluation metrics for this research consists of several measures. The first one is the calculation of annual profit that indicates whether the system is making a profit or incurring a loss in a year. The profit calculation is described in Eq. 1. The lowest profit or the maximum drawdown indicates how much the agent loses when opening a position. It is calculated by finding the minimum value of Eq. 1 or the maximum value of Eq. 2. Furthermore, the Sharpe ratio is utilized to indicate whether the obtained profit is consistent.

The second metric is the average holding time which is the duration of time that the agent holds a position. This calculation measures the length of time the agent holds a position, whether it results in a profit or a loss. Additionally, the average delta time which is the interval between the end of the previous trade position and the beginning of the next one is also calculated. The percentage of profitable positions indicates the number of profitable positions out of all opened positions. The standard deviation of holding time is also calculated to know if the agent closes the position flexibly, hence the holding time value should vary.

# 6 Experiment

## 6.1 Experiment Scenarios

The aim of the experiment was to address the research question. The independent variable, which referred to the time window of holding a position and not opening a position (delta time), was manipulated to observe its impact on the dependent variables which were profit and drawdown. The first experiment used a fixed time window of 4 h, which was similar to the window data described in Sect. 3.2, as a baseline to compare with other experiments. The second experiment completely removed the time window from the system to see the effect without any time window involved. In the third experiment, a threshold value of one day was introduced to ensure that the holding position did not exceed one day despite the flexible position and to force the agent to open a new position at least once per day. In the fourth experiment, the threshold values of delta time were gradually increased from 30 min, 1 h to 4 h to allow the agent to open a position after
a longer range from the previous closed position. It was expected to open the profitable position once the agent decided to open the position. Furthermore, target pips were added to make the agent close the position based on how much profit or loss the agent had obtained within a day.

These experiments as listed in Table 2 aimed to determine whether the system was able to achieve more profitable while reducing loss results. In each experiment, the values of the other components of the state used were constant as defined in Table 1, in addition to the values of holding time, delta time, and target pips. The starting balance for this experiment was set at $1,000 USD. If the calculated balance reached 0 during testing, the system terminated. Furthermore, if the testing simulation reached one year, any opened position closed.

<table><caption>**Table 2.** Experiment scenarios</caption><thead><tr><th rowspan="2">Scenario</th><th rowspan="2">Method</th><th colspan="2">Parameter</th></tr><tr><th>Holding and Delta time</th><th>Target pips</th></tr></thead><tbody><tr><td>1</td><td>Reinforcement Learning with fixed time window</td><td>4 h</td><td>—</td></tr><tr><td>2</td><td>Reinforcement learning without time window</td><td>—</td><td>—</td></tr><tr><td>3</td><td>Reinforcement Learning with one-daye threshold value</td><td>oneday</td><td>—</td></tr><tr><td>4</td><td>Reinforcement Learning with target delta time and target pips</td><td>30 min, 1 h, 4 h, one day</td><td>3, 10, 17, 27</td></tr></tbody></table>

## 6.2 Experiment Results

The data used in this study includes 446, 458 records spanning from 2017 to 2022. The testing simulation was performed on a yearly basis and the average result is calculated by taking the average of the results for each year. Experiment results are displayed in Table 3 and include metric information as outlined in Sect. 5.5.

**The First Experiment.** It involved the introduction of a time window to see if it had any effect on the results. This experiment achieved the lowest annual return, with a two-year simulation resulting in a loss in the total balance. The average profit was higher than the average loss, but the average highest profit was lower than the average maximum drawdown. This experiment had the highest Sharpe ratio value, due to the fixed time window leading to consistent profit. The average holding and delta time were 48 bar data or 4 h, as the fixed time window was set, leading to a low standard deviation of holding time since the agent did not close positions in a flexible manner. However, this experiment had the highest percentage of profitable positions among all the experiments.

**The Second Experiment.** It aimed to assess the impact of completely removing the time window. The results showed that this experiment achieved the highest annual return,
<table><caption>**Table 3.** Experiment results</caption><thead><tr><th>Scenario</th><th>Annual return</th><th>Avg profit</th><th>Avg loss</th><th>Avg Highest profit</th><th>Avg Max. Drawdown</th><th>Avg Sharpe Ratio</th><th>Avg delta time</th><th>Avg holding time</th><th>Avg Std</th><th>Avg profitable position</th></tr></thead><tbody><tr><td>1</td><td>36.2%</td><td>$14.2</td><td>-$13.95</td><td>$96.68</td><td>$124.845</td><td>1</td><td>48</td><td>48</td><td>0.6</td><td>50, 6%</td></tr><tr><td>2</td><td>654.8%</td><td>$4.5</td><td>-$3</td><td>$140.68</td><td>$74.34</td><td>0, 62</td><td>2</td><td>6</td><td>9.4</td><td>46%</td></tr><tr><td>3</td><td>440.4%</td><td>$2.7</td><td>-$2</td><td>$156.5</td><td>$89</td><td>0, 48</td><td>1</td><td>3</td><td>3.4</td><td>47.25%</td></tr><tr><td>4</td><td>42.1%</td><td>$2.8</td><td>-$2.4</td><td>$185.38</td><td>$43</td><td>0, 46</td><td>9</td><td>5</td><td>15.17</td><td>47.45%</td></tr></tbody></table>

likely because the average profit was higher than the average loss and there was a large gap between the highest profit and maximum drawdown. Additionally, the Sharpe ratio value was high enough to indicate that the profit was consistent. Although the average holding time and delta time were short (30 min and 10 min respectively), the longest holding time was 472 bar data which exceeded one day. The standard deviation of holding time was also high, indicating that the agent closed positions in a flexible manner. However, this experiment had the lowest percentage of profitable positions among the experiments.

**The Third Experiment.** It involved adding a threshold value of one day to the holding and delta time, making it a non-fixed time window that was part of the state component. This experiment resulted in the second highest annual return. The difference between average profit and average loss was small, with the average highest profit higher than the maximum drawdown. The average Sharpe ratio was lower than the first experiment, indicating that the profit was not as consistent. The average delta time was short, with the agent waiting only 5 min to open the next position, and the average holding time was 15 min (3 bar data). The standard deviation of holding time was lower than the first experiment but higher than the second experiment, meaning the holding time value varied but still in a flexible manner. The highest holding and delta time were 127 and 73 respectively, indicating success in restricting the agent to close positions within a day and open at least one position a day. The percentage of profitable positions was higher than the first experiment.

**The Fourth Experiment.** It was designed to increase the delta time with a gradient of threshold values and added the target pips alongside the one-day threshold value for holding time. This was motivated by the short delta times seen in the first and third experiments. The addition of the gradient threshold value for delta time was successful in increasing the delta time, but the target pips did not result in longer holding times, as the holding time was similar to the other experiments without fixed time window. The standard deviation value was the highest among all the experiments, indicating a high level of variation in the holding time value. The highest holding time was 475 bar data, likely due to the target pips allowing the agent to hold positions despite the one-day threshold for holding time. The highest delta time was 158 bar data. The difference between the average profit and average loss was small, resulting in a lower annual return than other experiments without fixed time window but still profitable due to the large gap between the highest profit and maximum drawdown.
## 6.3 Analysis Data

The second experiment, which removed the time window, resulted in a significant increase in profit, with an annual return of 654.8% compared to the baseline's 5.1% annual return [3]. The loss was also reduced, as the first experiment recorded a $74.34 loss or 7.4% average maximum drawdown, which was lower than the baseline's 11.89%. This might be due to the flexible close position, as shown in the standard deviation of holding time value, allowing the agent to close the position at a flexible time rather than a fixed time window.

Moreover, the implementation of threshold values and target pips also led to an increase in profit and a decrease in loss. The lowest maximum drawdown was obtained by implementing target pips. The threshold value and target pips did not restrict the closing position as the agent could open and close the position flexibly, as shown by the high standard deviation values in the third and fourth experiments. Contrary to the assumption, the removal of the fixed time window did not result in inconsistent profits. The first experiment, without a time window, achieved a Sharpe ratio of 0.62, which was slightly lower than the baseline's 0.63. However, Sharpe ratios in the third and fourth experiments were low, indicating that the profits were not as consistent.

The use of higher timeframes, such as 1-h and 4-h, was also beneficial. The profitable positions came from states where the bar trend of 5-min was the same as the bar trend of 1-h and 4-h timeframes. This indicates that the trend was occurring in both lower and higher timeframes and not just the volatility of the 5-min timeframe, leading to potential profits.

# 7 Conclusion and Recommendations

## 7.1 Conclusion

Removing the fixed time window for opening and closing positions in the forex market is a challenging task because it may result in inconsistent profits. However, removing this window may also lead to increased profits and reduced losses. This study proposes a reinforcement learning system with LSTM that enables flexible opening and closing of positions, eliminating the need for a fixed time window. The higher timeframe is also incorporated into the state design.

The results indicate that the proposed system without a time window improves profits by 654.8% and reduced losses by 7.4% compared to the system with a fixed time window of 4 h and the baseline paper. This is due to the flexibility of the proposed system in opening and closing positions, leading to increased profits. Furthermore, the fear of inconsistent profits due to the removal of the time window is disproven, as the proposed system still achieved consistent profits, reflected in the Sharpe ratio value of 0.62, which was only a 0.01 difference from the baseline paper.

Applying a fixed time window is found to be useful in obtaining consistent profits, as reflected in the Sharpe ratio value of 1. However, this is not suitable if the trader wants to obtain bigger profits instead of consistent profits, as it gives less profit compared to other experiments without a fixed time window.
## 7.2 Recommendations

This research has the possibility of being applied to different currency pairs, although it requires training on other pairs. It can also be improved by adding fundamental factors to determine forex market actions, due to their strong correlation.

## References

1. Hayley, S., Marsh, I.W.: What do retail FX traders learn? J. Int. Money Financ. 64, 16–38 (2016). https://doi.org/10.1016/J.JIMONFIN.2016.02.001

2. Moeeni, S.: Is It Necessary to Restrict Forex Financial Trading? A Modified Model 1 Introduction, vol. 13(1), pp. 63–80 (2019)

3. Shavandi, A., Khedmati, M.: A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets. Expert Syst. Appl. 208, 118124 (2022): https://doi.org/10.1016/j.eswa.2022.118124

4. Chakole, J.B., Kolhe, M.S., Mahapurush, G.D., Yadav, A., Kurhekar, M.P.: A Q-learning agent for automated trading in equity stock markets. Expert Syst. Appl. 163, 113761 (2021). https://doi.org/10.1016/j.eswa.2020.113761

5. Huang, C.Y.: Financial Trading as a Game: A Deep Reinforcement Learning Approach, pp. 1–15 (2018). http://arxiv.org/abs/1807.02787

6. Carapuço, J., Neves, R., Horta, N.: Reinforcement learning applied to Forex trading. Appl. Soft Comput. J. 73, 783–794 (2018). https://doi.org/10.1016/j.asoc.2018.09.017

7. Zarkias, K.S., Passalis, N., Tsantekidis, A., Tefas, A.: Deep reinforcement learning for financial trading using price trailing. In: ICASSP, International Conference on Acoustics, Speech, and Signal Processing Proceedings, vol. 2019, pp. 3067–3071 (May 2019). https://doi.org/10.1109/ICASSP.2019.8683161

8. Pring, M.J.: Technical analysis explained : the successful investor's guide to spotting investment trends and turning points, p. 797

9. Treleaven, P., Galas, M., Lalchand, V.: Algorithmic trading review. Commun. ACM 56(11), 76–85 (2013). https://doi.org/10.1145/2500117

10. Fundamental analysis in the multi-agent trading system | IEEE Conference Publication | IEEE Xplore. https://ieeexplore.ieee.org/abstract/document/7733396 (Accessed 02 Feb 2023)

11. Wanniarachchi, H.J., Rathnayake, R.M.S.J.K., Thilina, S.G.I., Ganegoda, G.U., Manawadu, I.: Macroeconomic event base expert advisor for forex trades: through algo trading. In: 21st International Conference on Advances in ICT for Emerging Regions, ICter 2021 - Proceedings, pp. 189–194 (2021). https://doi.org/10.1109/ICTER53630.2021.9774804

12. Salkar, T., Shinde, A., Tamhankar, N., Bhagat, N.: Algorithmic trading using technical indicators. In: Proceedings of International Conference on Communication, Information and Computing Technology, ICCICT 2021 (2021). https://doi.org/10.1109/ICCICT50803.2021.9510135