Verification and validation techniques for streaming big data analytics in internet of things environment

IET NetworksVolume 8, Issue 3 p. 155-163 Special Issue: Edge Network Computing for Big Data Systems over Internet of ThingsFree Access Verification and validation techniques for streaming big data analytics in internet of things environment Aparna Kumari, Aparna Kumari Department of Computer Science and Engineering, Institute of Technology, Nirma University, Ahmedabad, Gujarat, IndiaSearch for more papers by this authorSudeep Tanwar, Corresponding Author Sudeep Tanwar sudeep149@rediffmail.com Department of Computer Science and Engineering, Institute of Technology, Nirma University, Ahmedabad, Gujarat, IndiaSearch for more papers by this authorSudhanshu Tyagi, Sudhanshu Tyagi Department of Electronics and Communication Engineering, Thapar Institute of Engineering and Technology, Deemed University, Patiala, Pb, IndiaSearch for more papers by this authorNeeraj Kumar, Neeraj Kumar Department of Computer Science and Engineering, Thapar Institute of Engineering and Technology, Deemed University, Patiala, Pb, IndiaSearch for more papers by this author Aparna Kumari, Aparna Kumari Department of Computer Science and Engineering, Institute of Technology, Nirma University, Ahmedabad, Gujarat, IndiaSearch for more papers by this authorSudeep Tanwar, Corresponding Author Sudeep Tanwar sudeep149@rediffmail.com Department of Computer Science and Engineering, Institute of Technology, Nirma University, Ahmedabad, Gujarat, IndiaSearch for more papers by this authorSudhanshu Tyagi, Sudhanshu Tyagi Department of Electronics and Communication Engineering, Thapar Institute of Engineering and Technology, Deemed University, Patiala, Pb, IndiaSearch for more papers by this authorNeeraj Kumar, Neeraj Kumar Department of Computer Science and Engineering, Thapar Institute of Engineering and Technology, Deemed University, Patiala, Pb, IndiaSearch for more papers by this author First published: 01 May 2019 https://doi.org/10.1049/iet-net.2018.5187Citations: 48AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract With an exponential growth of raw data generated from different sensors, actuators, and mobile devices, data analytics is becoming a challenging issue keeping in view of the heterogeneity of the data generated. The traditional database systems are not able to handle this huge amount of data. Current research and development are mainly focused on the analytics of this big data generated from different devices and overlook the difficulty of its secure streaming. In this study, the authors analyse and provide an overview on how to handle secure streaming data generated from different devices. It includes the major threats and risks, introduced during big data processing in Internet of things environment. Moreover, they analyse the existing security approaches and highlight emerging challenges required to process the heterogeneous big data in a secure manner for IoT applications, such as denial of service, malware, and phishing. The architectural details and security approaches required in each phase of big data processing life-cycle are explored in detail. Finally, various research challenges along with a case study are discussed and analysed. 1 Introduction Rapid growth and continuous usage of the smart Internet of thing (IoT)-based devices, a huge amount of data is being generated in the past few years, which may be called as big data (BD) as it varies with all essential V's. These devices perform the task as per the application and used protocols without human intervention [1]. Storage of this heterogeneous raw data is a key challenge; however, use of cloud has solved this issue up to a significant level. As the devices are heterogeneous and continuously generating the data, so handling such a huge proliferation is the biggest challenges in front of the research communities. In addition to storage, another major challenge is the security of BD. According to Interactive Data Corporation [2], the BD describes a new generation of technologies and architectures. It is designed to extract value from large volumes of a wide variety of data, by enabling the high-velocity capture, discovery, and analysis. Gantz et at. [3] predicted that data may increase from 130 exabytes to 40,000 exabytes by 2020 due to growing use of applications such as scientific research, finance, government, Internet search, document, photography, audio, video files, business informatics, social networks, mobile phones, sensor networks, logs, and click streams. This intense growth in volume, variety, and velocity of data is difficult to handle using traditional data processing applications. BD promotes scientific and economic research, which transforms scientific methods into traditional models and creates a new opportunity for real-time analysis [4] as per applications. To improve the internal decisions, many establishments at the global level collect and analyse business process data. Nevertheless, exploring the values of BD increases the risks of security and privacy. For instance, Google learns users browsing habits, Amazon monitors customers shopping inclinations, Twitter knows what's on user's minds, and Facebook seems to catch all data of the user's along with user's social relationships. Mobile operators not only know the contact numbers of the users but also have their personal information. BD provides valuable insights by analysing these personal databases [5]. BD security solutions are used to increase the reliability, security, and safety of a distributed system. BD privacy focuses on the protection from unwanted inference and unauthorised access of data [6]. Moreover, it is a source of information at the base of accurate and robust security solutions. Furthermore, it contains sensitive data which needs to be protected from illegal access. Apparently, challenges of security and privacy arise while the value is derived from BD. Hence, there must be a balance against the additional societal value of BD [4]. Fig. 1 presents the basic architecture of streaming the BD in IoT systems. The collection of high volume data from a group of transactions over a fixed or pre-defined period is known as batch processing. Structured, semi-structured, and unstructured data can be stored separately using the batch processing. These are useful for event-based processing but not for low latency applications such as Healthcare and On-line gaming. In Healthcare systems, where priority is given to the cost and complexity, batch data processing is useful. On the other hand, processing over small data over short or recent time is known as stream processing. Computations performed by such a technique seem as real-time processing, as it takes less time for processing. This method maintains a high level of accuracy in parallel with stream processing, which is required for time responsive data processing. To use this method for BD applications, strong mathematical models are required to perform continuous streaming analytics. Stream processing supports the healthcare system to make accurate decisions for doctors or medical specialists. Consequently, an immediate and effective action can be taken to maintain the health of the patient [7]. Moreover, if the data is collected from the wearable smart devices then medical treatment of the patient can be done at home which avoids unnecessary hospitalisation [8]. Fig. 1Open in figure viewerPowerPoint Basic architecture of BD streaming in IoT environment 1.1 Existing surveys To date, several surveys have been conducted, which explored various aspects of BD technology [4, 9-15]. However, to the best of our knowledge, most of these surveys focused on analytics in BD and there is no survey that considered the difficulty of BD secure streaming in IoT environment. Our survey spans over last five years (2014–to date). Matturdi et al. [4] outlined the benefits and challenges of security and privacy in BD. Ida et al. [9] conducted a comprehensive survey on security in IoT in context of ehealth and clouds. Nawir et al. [10] surveyed the taxonomy of security attacks in IoT and implementation for secure solutions. Furthermore, Chandra et al. [11] discussed some of the frameworks and algorithms related to BD security. Other BD security surveys include application-wise in depth coverage of security in smart city [12], block chain-based security solutions for IoT [13]. Furthermore, Mohammadi et al. [14] conducted a survey on streaming analysis and their preferences for deep learning technologies whereas Ge et al. [15] performed integration of BD and IoT and also compared state-of-the-art security solutions in BD. We conclude from the aforementioned discussion, most of the recent surveys on streaming BD in IoT applications have inclined towards specific topics of the BD. Table 1 presents a summary of these surveys and their differences with the proposed survey. Table 1. Comparison of the existing surveys with the proposed survey Year Authors Objective(s) of the survey Merits Demerits 2014 Matturdi et al. [4] highlights the benefits and challenges of security and privacy in BD processing and managing of BD security issues with respect to streaming of BD were not highlighted 2016 Ida et al. [9] security of IoT in the context of eHealth and clouds layers of IoT vulnerability were covered role of BD streaming was lagging and its security issues were not highlighted 2016 Nawir et al. [10] to propose the taxonomy of security attacks in IoT comprehensive review of layered based attacks was given secure streaming of BD in IoT applications is still a big challenge 2017 Chandra et al. [11] to explore the frameworks and algorithms for BD security current BD scenario was discussed in detail with key security issues only topic oriented BD security was explored 2017 Joshi et al. [12] to explore issues of BD security and privacy application-wise in depth coverage of security and privacy key issues for the development of smart city role of IoT and BD streaming was missing 2018 Khan et al. [13] IoT security through block chain state-of-the-art IoT security issues and solutions were discussed no parameters explored to secure the BD and limited coverage of IoT security and privacy issues only 2018 Mohammadi et al. [14] streaming analytics and deep learning for IoT BD IoT applications which used deep learning and key research challenges were covered role of security in BD streaming was lagging 2018 Ge et al. [15] integration of BD and IoT comparison of the BD technologies across all state-of-the-art IoT domains security and privacy issues were not explored — proposed verification and validation techniques for streaming BD in IoT secured streaming, detailed case study, and explored emergent challenges to securely process the heterogeneous BD — 1.2 Motivation and contribution Although several research proposals exist in the literature addressing various issues of BD and IoT, security and privacy of data collection, storage, processing, and analysis of BD streaming have not been exploited to its full potential. In this study, we investigated the security and privacy issues on the above-mentioned parameters for BD streaming. The following are the objectives of this study. To develop architecture covering security by function and by platform required for BD streaming in IoT environment. To perform a comparative analysis of several existing approaches to the security of BD streaming with respect to different performance evaluation parameters such as access control, classification, security policies, and risk analysis. A case study is presented to demonstrate the streaming BD in IoT environment. 1.3 Organisation The rest of the paper is organised as follows. Section 2 describes the BD streaming with architecture for IoT and background of BD streaming. A taxonomy for BD streaming-security based on IoT is presented in Section 3. Section 4 covers the case study on Healthcare 4.0. In Section 5, research challenges are discussed and finally, the paper is concluded in Section 6. 2 Background for BD streaming The IoT is a network that interconnects various heterogeneous low power devices [16]. The huge amount of data generated from IoT devices can be viewed as BD which serves billions of users globally. The popularity and growth of IoT have come at an expensive cost due to the loss of resources and information caused by security threats and attacks. In recent years, security breaches are increased by cyber criminals constantly exploring new ways to by-pass security solutions to get prohibited access to data and systems [17]. Some of the security breaches are as follows: Spamming: In this technique, unsolicited bulk messages are sent to multiple receivers [18]. Access filtering, computing, and content are major anti-spam techniques used to handle this issue. Access filtering detects the spam based on the email addresses and internet protocol (IP) addresses. Munging the email addresses makes it unusable for spammers, e.g. xyz@gmail.com munges as xyz at Gmail dot com. The content filtering identifies predefined text patterns in emails to detect the spam [19]. Malware: It is software programmed to propagate and perform malicious activities such as worms, viruses, and Trojans. Worms are self-propagating, viruses require human intervention for propagation and Trojans are non-self-replicating. The harm from malware includes corruption of BD or operating system, theft of personal credentials or hard disk space, installation of spyware etc. [20]. Denial of service (DoS): A system or other network resource, which is inaccessible to its envisioned users is known as DoS. It is thrown by a large number of dispersed hosts such as the botnet. Various techniques such as firewalls, intrusion detection, and puzzle solution are common to avoid DoS attacks [20]. In this case, the BD generated from the system attacked by DoS can be missing for processing as it is unavailable for user access. Search poisoning: It is the fraudulent use of optimisation techniques for search engines to falsely increase the page rank [4]. Normally, regular search keywords are used to direct users which opened the websites illegally [21]. The very first issue of search poisoning was reported in 2007 [22]. Phishing: It acquires confidential user data by mirroring e-communication using the web and email spoofing. In web spoofing, dishonest websites mimic legitimate web pages to deceive users into entering confidential data. In email spoofing, fake emails direct users to fraudulent web pages which trap the user confidential data. Various anti-phishing solutions were proposed to counter the security threat [23]. Botnets: It is a network of malware-infected cooperated computers managed by a rival. The attackers use bot software furnished with the control system to control these bots and integrated command to group them into a network. The BD generated from Botnets can be malicious for streaming or processing [24, 25]. Website threats: It refers to exploiting vulnerabilities in an authentic website. Search result redirection, structured query language (SQL) injections and malicious advertisements are some of the techniques used to infect the authentic websites [26]. To design and implement an access control, authorisation & authentication, and intrusion detection techniques which can work in heterogeneous IoT architecture for BD streaming is a key challenge. For authentication, biometric-based verification in IoT environment can be used. There are issues associated with the intrusion detection in geo-distributed, large-scale, and high mobility IoT environment. One of the possible solutions is public and private key setups and reliable environments for execution such as a trusted execution environment [26]. End users must be ready with reserve plan for failure of the service platforms, sensors devices, networks, and applications. Considering the above-mentioned facts, there are lots of scope to works on verification and validation techniques for BD streaming. In the next section, we discuss the broad taxonomy of verification and validation techniques for BD streaming in IoT. 3 BD streaming security in IoT environment: a taxonomy A detailed taxonomy of BD streaming is presented in Fig. 2 which includes various parameters and pre-existing approaches. For better understanding, we have divided the taxonomy into two parts: (a) security by functions and (b) security by platforms. Next subsection describes the various parts of taxonomy in detail. Fig. 2Open in figure viewerPowerPoint Main taxonomy of security on BD streaming in IoT environment 3.1 Legal issues and compliance Precise laws and regulations are essential and must be available to safeguard sensitive data of users prior to access it. Every country has its own laws and regulations to protect their database. Monitoring of company staff activities is not permissible by the third party or parent company in many countries. Nevertheless, electronic monitoring is permitted under exceptional conditions [27]. Consequently, in most of the situations, laws and regulations are sufficient to protect users private data but for BD [28], it needs careful attention to protect the critical data of users. Data governance bodies have to complete their homework to control and authorise to handle the BD issue. Therefore, timely enforcement of the laws, rules, transparency, and accountabilities of the user's information to achieve business objectives [29] is necessary. The BD governance affects those applications which consume enormous amounts of streaming data from IoT devices [30]. BD security involves leverage of data by multiple objectives such as smart grid having access to large customers and their utility database. Consequently, acceptable and clear data policy with respect to its data type and accessibility must be defined [31]. Most of the time third parties provide the marketing to the large companies against monetising. Furthermore, BD provides noteworthy opportunities to service providers by creating valuable information. Though laws, guidelines, ethics, and frameworks make a balance between risk and value and deliver better data management. IoT, Cloud, and BD governance endorses the use of several policies together with different constraints, which limited the access to underlying resources. Moreover, espousing governance practices maintain a balance between risk and value for making the best use of the value of several applications such as smart grid and healthcare [27, 32]. 3.2 Threats and risks Threats detection and risk assessment are two major criteria to provide security to the BD generated from IoT sources. 3.2.1 Risk assessment Risk assessment is a combination of risk analysis and its evaluation [33]. It provides an organised way for existing BD security risks and their consequences with an approach to deal with them [34]. Risk assessment includes the risks associated with different types of platforms such as application programs, operating systems, people, networks, and processes along with interdependencies. It is a crucial task as the risk assessment is a costly affair in organisations. (a) Risk analysis: It recognises the valuable assets, their vulnerabilities, and discloses threats that may take advantages of vulnerabilities. Then, evaluate the possible damage and probable losses caused by identified risks. The risk analysis steps are as follows: Resource identification and valuation: Identify and evaluate the assets/resources, which are valuable to the organisation. Risk identification: It identifies the relationship between vulnerabilities and threats. Vulnerabilities can be identified based on how critical is the information assets and the availability of resources [35, 36] using different methods such as automated vulnerability scanning tool, security testing & evaluation, penetration testing, and code review. Risk measurement: It includes the model to measure risk, which specifies the relationship between various risk factors such as resource value, threat impact, and vulnerability effect [37]. Once the risk is analysed, then it needs to be evaluated by the organisation. (b) Risk evaluation: It is a process to rate the risk exposures on a scale and determine the significance of each risk. Then, apply the proper approach to address and manage the risk [36]. Moreover, the risk is a priority based on various factors such as legal, regulatory, financial or reputation impact [33]. There are four stages to address the risk: Accept: The organisation understands the risk and accept it. Avoid: The organisation avoids the risk based on its assessment. Transfer: All or a part of the risk can be shifted to another party. Mitigate: Control and limit the risk and its consequences are lower than the acceptable limit. In the risk assessment process, it is important to identify planned safeguards, failing which results in an inaccurate risk assessment that may put the organisation in danger and increase unnecessary costs [38]. 3.2.2 Threats detection The concept of BD security is expansive, and it can be categorised into active and passive sources in the IoT system [39], which also attacks and affects the availability of the IoT system [10]. (a) Active attacks: The active attacks break or escape the security and privacy feature of the data communication between different devices. This mess up the communication among the devices of the entire network. For example, person A is replying a message to person B, but the message is expected from person C, here, person A becomes the active attacker [40-42]. Active attacks relating to real-time data can include Authentication data such as Login ID and password. Digital certificates. One-time passwords for online access. Biometric identification such as fingerprint, handwriting recognition, face recognition, voice recognition etc. Knowledge-based questions like ‘what is your pet name?’ Social media data such as Facebook and Twitter. (b) Passive attacks: This type of attack involves snooping and monitoring of the communication in the IoT system. They do not know about a user and do not consent, also disturb the communication. They only use or learn the information from the communication. For example, person A sends a message to a person B this message can be read by person C. Hence, person C is called a passive attacker [41, 43]. Passive data sources from the IoT system can include Mobile-based data such as network location, global positioning system location, and wireless application protocol data. Computer-based data such as users clicking patterns, geographical IP location, keyboard typing, computer security certificates, and wireless application protocol data. Human resource data such as the privilege of the user and organisation role. Travel data such as journey, travel patterns, and destinations. User's physical data such as time and location of physical access of the network. Data from external sources such as external threats and rogue IPs. Security information and event manegement (SIEM) data such as application access data, network logs, and threat database. BD security analyses on active and passive IoT sources collectively provide a complete view of system activity such as alerting an abnormal behaviour of a user's access pattern of the system. Prevention techniques such as lock the accounts, modify network settings, confinement, multiple authentications, and generation of an alert message on an on-going fraud [22] can be applied. 3.3 BD protection Handling the secure data is quite different from handling plaintext data, where the authentication and confidentiality are taken into consideration. Authentication: Some BD applications require authentication, which protects the BD from modification. Moreover, it deals with a certification authority and user authentication. Confidentiality: The confidentiality is related to data encryption and decryption. Alouneh et al. [44] proposed an effective classification approach for BD security, where security to BD is provided based on different BD classes such as structured, semi-structured, and unstructured data. Moreover, other security factors such as an access control list and DoS protection also need security attention. Fig. 3 shows the BD classification to provide security against data streaming of different classes such as structured, semi-structured, and unstructured data. In this architecture, first, incoming data is categorised. Then, check the security services such as encryption to be applied or not, whereas, authentication is required on the processed data. The next step takes proper action based on the security service applied to the processed data over the network such as virtual private network, local area network, wide area network and so on. Finally, analytic is based on ten V's as explained briefly in the next subsection. Fig. 3Open in figure viewerPowerPoint BD classification 3.3.1 BD classification The arrival of the Internet, IoT, and social networks provide an immense increase in the collection of gigantic real-world data. The huge data generated from different sources contain numerous forms of data such as text, audio, video, numeric data, and logs. Due to which, the BD is not limited to only three V's [45], i.e. volume, variety, and velocity. Moreover, the BD classification approach should take the ten V's, which are defining several properties of BD with respect to Healthcare 4.0 system. (a) Volume: Continuous data collection is providing a huge volume of data, which includes routine records of doctors, supporting staff, nurses, and patients in a hospital. Moreover, the collection and storage of regular monitoring of critical areas, patients, and their electronic health records (EHRs) also affect the volume of data. This property is used to calculate the storage requirements. (b) Variety: Types of data such as 2D images, 3D images, text records, voice, and video recordings during a surgery brings variety of BD for Healthcare 4.0 system. In general, a variety of data is available in the form of structured, semi-structured, and unstructured format for healthcare, which is huge and challenging too. Moreover, a collection of health-related data from different sources such as sensors, cameras, smart devices, and social media also define the variety of properties. (c) Velocity: The rate at which data is being generated and captured defines the velocity property of BD. With an exponential growth of smart devices (those are used in the healthcare industry), potential data is being received at a very faster rate. Even social media is also generating health-related external data at a faster rate. Therefore, velocity is one of the challenging property in Healthcare 4.0. (d) Variability: Types of inconsistencies observed in BD are known as variability. This property is used to locate the anomaly or false alarm which may affect the health condition of the patient. Variability can also observe an inconsistent rate at which potential data is loaded to the database. (e) Vulnerability: This property of BD is directly related to the security of the Healthcare 4.0. By knowing the vulnerability, one can avoid the possibility of an attack in the healthcare system. (f) Volatility: It is related to the two different properties velocity and volume of stored data which signifies the importance of data before it seems irrelevant or not useful for any purpose. Therefore, knowing the volatility helps the designing of a storage system for Healthcare 4.0 with proper estimation of cost and complexity. (g) Validity: The stored data which can be used for the Healthcare 4.0 system needs to check the validity of data. Amount of accuracy and correctness of essential data can be measured on the basis of the validity of BD. (h) Veracity: It shows the amount of reliability in terms of sources of data or can also be measured as the confidence in the data. Consider a data set of Healthcare, where EHR of different patients is maintained for the last ten years. Now, researchers are interested to know the number of successful surgeries performed by senior doctors in the last two years. Which methodology did they use in the surgery? What was the usual team size involved in the surgery? Whether the recorded data was secured or not? The aforementioned issues can be solved using the veracity property of BD. (i) Visualisation: Visualisation of BD is the most critical challenge, in addition to variety and velocity because the Healthcare 4.0 systems maintain the scalability, functionality, and decrease the response time. (j) Value: Without value, any organisation, system or business may fail. In Healthcare 4.0 system, it is necessary to give the value to its custodians such as doctors, staff, nurses, and patients. A regular monitoring system and systematic feedback may help to maintain the value. In short, the data drive with meaningful information is known as value. Table 2 provides the detailed comparison of existing approaches used to handle the secure streaming BD in IoT with reference to paramete