## 2 Paper Research and Evaluation
### Citation Network  
To create the citation network, data from the references of each document in the search were used, making each publication a node in the network. Two nodes are linked with an edge if one of the corresponding documents cites the other one. The resulting citation network comprises 485 nodes (documents) and 863 edges (citations). It is shown in Figure 7, where colour indicates the year of publication of the document pertaining to that node, and size indicates the number of times it has been cited:  
The most cited document in the network was by [PERSON] et al. (2015) [PERSON] et al. (2015), cited 25 times locally (within the citation network) and 12,691 times globally (in the database search). They used a Deep Q-Network to demonstrate its capabilities against previous algorithms in beating Atari 2600 games. The second most cited document was by [PERSON] et al. (2016) [PERSON] et al. (2016), cited locally 14 times and globally 1354 times. They developed a method to learn certain deep convolutional neural network policies to determine whether it is better to train the perception and control systems jointly end-to-end or train each component separately.  
Figure 6: Venn diagram of DRL approaches occurrences in the research query.  
Figure 7: Citation network with colour and size legends.  
The oldest document in the network was by [PERSON] et al. (1930) [52], in which they discussed the Brownian motion. The second oldest document was by [PERSON] et al. (1985) [53], published in 1985, in which they discussed dynamic obstacle avoidance for manipulators with motion control and multiple goals.  
To analyse the appearance of keywords in the citation network, the following search queries were performed and grouped accordingly: (i) _manufactur_* or _automat_* or _\"production system\"_, (ii) _robot_*, (iii) _*polic_* and (iv) _\"reinforcement learning\"_. Note that the asterisks are used to replace multiple characters anywhere in a word. For example, the query _\"_\"polic\" would return documents containing any of the following terms: policy, policies, on-policy and off-policy, among others.  
The results can be seen in Figure 8, where size indicates the number of times the pertinent document has been cited and colours are chosen based on the grouping of the Venn diagram:  
Figure 8: (**A**) Citation network coloured based on the appearance of selected words with size legend; (**B**) coloured Venn diagram of keywords. The asterisks are used to replace multiple characters anywhere in a word.  
Of the 485 documents in the network, 402 satisfy any of the previous search queries. Overall, 71 documents (14.64%) belong to the first group, 267 (55.05%) to the second group, 162 (33.40%) to the third group and 297 (61.24%) to the fourth group. Only 17 documents (3.51%) are part of the four groups simultaneously. As can be seen, the documents from the original search query rely mainly on other documents that encompass robotics and reinforcement learning, meaning that the application of DRL in manufacturing processes is a relatively new topic that is being developed from the ground up with its base built upon previous works in robotics and reinforcement learning. This is also demonstrated by the fact that the original bibliographic export comprised mainly documents from 2016 onwards. Then, it is to be expected that in the following years, future work will be built upon today's works with exponential growth as new developments in DRL occur.