###### Abstract  
The IETF Routing Over Low power and Lossy network (ROLL) working group defined IPv6 Routing Protocol for Low Power and Lossy Network (RPL) to facilitate efficient routing in IPv6 over Low-Power Wireless Personal Area Networks (6 LoWPAN). Limited resources of 6 LoWPAN nodes make it challenging to secure the environment, leaving it vulnerable to threats and security attacks. Machine Learning (ML) and Deep Learning (DL) approaches have shown promise as effective and efficient mechanisms for detecting anomalous behaviors in RPL-based 6 LoWPAN. Therefore, this paper systematically reviews and critically analyzes the research landscape on ML, DL, and combined ML-DL approaches applied to detect attacks in RPL networks. In addition, this study examined existing datasets designed explicitly for the RPL network. This work collects relevant studies from five major databases: Google Scholar, Springer Link, Scopus, Science Direct, and IEEE Xplore(r) digital library. Furthermore, 15,543 studies, retrieved from January 2016 to mid-2021, were refined according to the assigned inclusion criteria and designed research questions resulting in 49 studies. Finally, a conclusive discussion highlights the issues and challenges in the existing studies and proposes several future research directions. 6 LoWPAN; Internet of Things (IoT); IPv6; Low Power and Lossy Network (LLN); Machine Learning (ML); Deep Learning (DL); RPL security and threats; Systematic Literature Review (SLR) 202222022220 benefits of IoT are nearly limitless, and its applications are transforming how we work and live by exchanging time and assets and opening up new prospects for growth and development [5]. Moreover, a recent forecast from International Data Corporation (IDC), a well-known provider of industry intelligence, predicts that by 2025, there will be around 41.6 billion connected IoT devices/things (a combination of sensors, machines, cameras, etc. ), generating around 79.4 zettabytes of data. The forecast was based on an analysis spanning the years from 2018 to 2025, and during that period, they expected IoT devices to grow at a Compound Annual Growth Rate (CAGR) of 28.7%. The various IoT sensors deployed in different environments are responsible for collecting data in the network and sending them to the backbone servers and control centers for further analysis to assist in decision making. The number of IoT-based applications has increased exponentially in recent years, and while many are still in the early research stage, many economically attractive application scenarios already exist that span several domains [6]. Some of these applications include smart firefighting for forest fire detection and personal protective equipment monitoring; smart manufacturing for monitoring air quality, temperature, and cyber-physical systems; and intelligent healthcare for detecting Ultraviolet (UV) radiation, monitoring patient conditions, and controlling emergency response vehicles. Since the IoT exchanges massive quantities of essential and sensitive data, the lack of security of those networks, especially involving security breaches or penetration, could lead to severe repercussions economically and endanger human lives [7, 8]. In IoT, the information is exchanged and routed among the linked devices through a specially designed network that supports IoT specifications. An example of such networks is Low power and Lossy Networks (LLNs), comprising a wide range of embedded devices, such as sensors and actuators. Those devices are the driving force behind the IoT, since they enable global connections to items that are not connected to the Internet. However, these embedded devices have a low power supply, small memory space, limited computing capabilities, and a short radio range. Hence, to enable communication among those appliances, the Internet Engineering Task Force (IETF) specified IPv6 over Low-Power Wireless Personal Area Networks (6 LoWPAN) to serve as an adaptation layer and effectively encapsulate long IPv6 headers in packets of 128 bytes [9, 10].