prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What is the definition of mobile device management (MDM)?

Read the following text and answer briefly.

Context 1: Hospitals as critical infrastructures has been historically dependent on various types of devices and equipment that are being revolutionized with digitalized solutions  The digitalization of conventional healthcare equipment is added with the new inclusion of numerous new devices for data collection, analysis, communication, and so on  All in all, the futures digital hospitals in 5G will be exponentially more data-dependent and digital-intensive  For that, this paper looks to theorize how the security scenario in a futures digital hospital would look like, and what relevant business possibilities could emerge for cybersecurity providers in the healthcare context  In this paper, we open up discussions on business possibilities relevant to Internet of Things-mobile device management for critical infrastructures such as future digital hospital  We apply business models as a conceptual lens to analyze how cybersecurity business could evolve for 5G enabled IoT-Mobile device management providers as a cybersecurity vendor 
The healthcare sector has progressed significantly since the introduction of Internet and proliferation of network technologies   Among many issues, the use of data, availability of data, data mass, and access control of data in healthcare remain are critical for keeping healthcare services trustworthy and secure for the end users, both hospital staff and the patients  Disruptions in healthcare services would have severe effects on people’s lives  However, as hospital managers and professionals need to design their data-dependent and digital-intensive networks in a manner, which is highly secure, they also should provide the basis for uninterrupted service for business sustainability  Security is often observed as a tradeoff between risk and business gains   Investing in security is important in order to secure business-critical systems and data for meeting business goals and eventually for creating competitive advantage [3, 4] 
Innovative technologies have the power to disrupt industries and prompt business transformation   The coming of fifth generation (5G) of telecommunications networks is seen to result in this kind of disruption  As we are gradually moving towards 5G, it is worthwhile to theorize how the security scenario in a futures digital hospital would look like, and what relevant business possibilities could emerge from cybersecurity in the healthcare context  From this perspective, in this paper, we open up discussions on business possibilities relevant to Internet of Things mobile device management for critical infrastructures such as future digital hospital  We apply business models as a conceptual lens to analyze how cybersecurity business could evolve for 5G enabled IoT device management providers as a cybersecurity vendor 
A futures digital hospital facility is envisioned to consist tech-aided advanced critical medical devices, intelligent information systems, digital communication tools, hundreds of handheld mobile devices (smartphones, tablets), wireless clinical wearables, in addition to thousands of smart IoT nodes [6, 7]  These devices should be fully integrated to improve staff productivity, hospital functions, patient safety and privacy, and, overall improve patient experience through secure and reliable healthcare services  However, inclusion of these various kinds of digital devices to the hospital context make the overall device network quite complex and heterogeneous   Thus, from a critical infrastructures view point, to manage, configure, update and secure the immense fleet of digital devices besides all the high-tech medical equipment, the futures digital hospital will need to redefine device management policy and services  
Mobile device management (MDM) systems are usually referred to “support centralized control of an entire fleet of mobile devices (smartphones and tablets) and mobile applications by applying and ensuring pre-defined configuration settings” [8, 9]  In the scope of this paper, we broadly use the term IoT-MDM to refer to a device management system that is capable of managing, configuring and updating both handheld mobile devices and IoT devices in combination in a centralized manner  We will briefly open up the concept of MDM and IoT-MDM 
The purpose of this study is to identify business potential for IoT-MDM service providers as cybersecurity vendor in the context of the future digital hospital  In doing so, we apply the concept of business model in order to make sense of a ICT-oriented business environment   Among various available conceptualizations, business model is considered as a boundary-spanning unit of analysis that explain the underlying business logic and the value creation and value capturing logic of an organization [11–15] 
Traditionally, device management has been associated with management and configuration of handheld mobile devices , thus, mobile device management (MDM)  Gartner  perceives MDM software to be a policy tool to configure and manage mobile handheld devices  They also mark that MDM services need to ensure security in reference to connectivity and content that is being transmitted  Along with surge of smart mobile devices, the Internet of Things (IoT) is growing large during the last few years and promises to flood the market with billions of devices in the coming years too   Zhang et al   states scalability, transparency and reliability as important issues that differentiates IoT from the conventional Internet  To that end, there are several IoT platforms available currently in the market for managing, updating and configuring IoT nodes, e g  IBM Bluemix, Cumulocity, ARM mbed OS, etc    However, the transition raises the question about the differences and similarities between MDM and IoT device management as approaches 
Takalo  marks MDM and IoT device management to be quite close on a conceptual level: both need solution for automated management of large device fleets consisting different form factors, device models, and operating system  Additionally, such systems conceptually needs to support various communication channels like: WiFi, cellular network and Ethernet  However, on practical level, MDM is more strictly controlled by operating systems and device vendors  IoT device management, on the other hand, is characterized by multiple operating systems, multiple hardware platform variations, non-complete standards, multiple communication methods and protocols  This agile approach allows the coexistence of various types of devices and nodes in the same environment  Recently, services are appearing where MDM and IoT device management are brought together under the same platform, which in a way reduces the complexity in device management and also improves the overall security of the system 
While we are still in early phase of mass IoT deployment and 5G deployment is approaching, it is important to revisit some of the key security threats that has been identified in recent literature relevant mainly to IoT  Zhang et al  [19, 21] identified privacy preservation as a critical issue for information security in IoT ecosystems  They further state that conventional naming, identification and authentication policies need to be improved and rather needs approaching differently  Farooq et al   look at the four layer generic IoT architecture consisting perception layer, network layer, middle-ware layer & application layer  Additionally, they offer a list of security challenges for each layer  Some of the key challenges are: unauthorized access, tag cloning, eavesdropping, spoofing, RF jamming, sinkhole attack, sleep deprivation attack, denial of service attack, malicious code injection, man-in-the-middle attack, spear-phishing attack, sniffing attack  Backman et al   state, a comprehensive security solution needs to address endpoint security, management and monitoring security, and secure data distribution and storage 
Ortbach et al   enumerate the drivers for adoption of MDM in organization through a quantitative analysis, reflecting three broad drivers: organization, environment, and technology  From organizational perspective, the company size, mobile IT usage, employee innovativeness with IT and BYOD (bring your own device) culture were identified as important drivers  From environmental viewpoint, regulations and other business partner influence are significant drivers for MDM adoption  Finally, from technology point of view, perceived security benefits and perceived cost of the service seems to affect the managerial attitude towards MDM adoption 
MDM systems are today a very common tool to manage users’ devices  With MDM, all mobile device types, tablets and PCs with typical operating systems can be controlled centrally   It is often thought that MDM can manage only mobile phones, but actually the MDM framework includes also users’ identities and profiles  This makes MDM a viable tool for organizations to manage their employees identities, user profiles, all devices, all applications and security controls under same system 
From an emerging technology perspective, SDN (software-defined network)/NFV (network functions virtualization) based 5G Slicing will challenge some of the traditional MDM features   Especially end-to-end security from device to IT cloud is difficult to realize with MDM  Of course, it is possible to force the use of VPN in mobile device with MDM, but many aspects of communications security will be still unsolved  Thus, 5G slicing provides new tools to control and manage the end-to-end communications flow with network functions (VNFs)  In the advent of IoT, this is particularly important since the billions of IoT devices of the 2020s will have only a minimal processing power and memory compared to the smart phones of today  These IoT devices may connect to network only once a month and communicate only with network edge cloud servers  Therefore, managing these new IoT devices cannot be done with conventional MDM systems of today 
Fortunately, many features of MDM can be provided by dedicated 5G slices and their VNFs  If e g  a IoT device does not have the latest anti-virus updates, the network slice may still provide the isolation and security controls so that the IoT device can send the metering data  Moreover, if the IP flow from the IoT device includes other than actual metering data, eg  due to malware in IoT device, this IP flow can be analyzed and filtered by slice specific VNFs before passing it to IoT could 
As a summary, it can be argued that SDN/NFV based 5G Slicing will provide new tools for security management, and, when combined with IoT-MDM system functionalities, together these can deliver a better device management framework for different kinds of user devices of the 2020s 
Hospital organizations are considered as critical infrastructure (CI) to nation states [3, 25]  As a critical infrastructure, hospital organizations are prone to security threats that can affect health policies, public health, healthcare services, surgical procedures, electronic patient records, patient privacy, doctor-patient communication, etc  Lehto and Ahokangas  notes, new technology (e g  next generation mobile networks, smart data storage, IoT) adoption of CIs increases the cybersecurity touchpoints and hence making the CIs more vulnerable  Broadly, from cybersecurity perspective, the hospital organizations in future will be vulnerable from management perspective (e g  organizing healthcare services, managing huge amount of patient data, clinical data, medication data, communication between health professional and patients etc ), healthcare service delivery perspective (e g  in an unwelcoming case of denial of service attacks in hospital context: like wannacry), network perspective (e g  security of the overall hospital network), and last but not the least from an individual privacy perspective (e g  individual patient records, healthcare professional logs, etc ) 
A futures digital hospital will consist various advanced technologies, such as critical medical devices, intelligent information systems and digital communication tools, which are fully integrated to improve staff productivity, hospital operations, patient safety, and the overall patient experience  Among them, many will be wireless mobile devices, wireless wearables and thousands of smart IoT devices for various kinds of measurements  
Innovations help transform healthcare in forms of advanced telecommunications techonology, new drugs and treatments like biological sensor pills or implants, new medical devices, social media interation, etc  [1, 6, 7]  In future, hospitals and surgical tools will be revolutionized with intelligence and connectivity i e  providing more assistive functions by sensors, processors, data collection, software algorithms and interfaces which are, for example, embedded in the tools that will support surgeon’s decision-making in action   Robotic-assisted Surgery (RAS) can be considered as a good example of such transformation  The rapid adoption of Minimally Invasive Surgery (MIS) and today’s information revolution not only improves the surgical outcomes and patient’s life but also changes the patient-physician relationship 
‘Smart’ i e  IoT-based technologies are going to intellectualize medical devices and service systems into Smart Hospitals  Smart hospitals extrapolate from totally digitalized and automated data collection, tracking and delivery between systems, devices, patients, and health professionals and organization   In future, it is also envisioned that virtual hospitals and personalized medication will become part of patient care  Patients will be able to visit virtual hospitals or e-clinincs for clinical purposes without the presence of health professionals, but procedures and communication are managed via remote management and telemedicine solutions  
Other features that futures digital hospitals will include: use of different AR (augmented reality) and VR (virtual reality) applications for healthcare related purposes  While VR applications are quickly evolving in the gaming industry, some applications are deemed suitable in hospital environment too  Patients can be introduced to hospital environment through VR solutions, which can reduce patient stress prior to surgical procedures   Another application of VR in healthcare has been identified as rehabilitation support for temporary physical disabilities and mental trauma  Further, AR solutions can help patients navigate within a large hospital complex 
The term, Business model, has had its root in information systems (IS) and information communication technology (ICT) since the late 1970s originating from business informatics  However, it came into management and strategic management literature as a research interest from the mid-1990s [29, 30]  Although business model had its roots in IS and ICT, the amount of research work on cyber security as a context is still quite negligible harnessing the potential of business model concept 
The concept of business models lies at the intersection of entrepreneurship and strategy, it can be observed as a bridge between abstract strategies and the practical implementation of strategic decisions and actions amidst the uncertainties of the modern business context [11–14, 31]  For instance, Zott and Amit  conceptualize business model as a ‘boundary-spanning’ set of activities aimed at creating and appropriating value  Morris et al   viewed the concept of the business model as a set of decisions related to the venture strategy, architecture, and economics of firm (value creation and capture) that need addressing to create sustainable competitive advantage in the chosen markets and specific contexts 
As a boundary-spanning unit of analysis, business models , connects an organization with its business environment, other organizations, customers, individuals and society as well; with the overall ecosystem at large [34, 35]  Trying to bridge business models and cyber security under current context, there are two core issues  First, as almost all of the entities operating within the digital sphere face multifaceted cyber threats, how can business model approach help organizations to respond to such situations? Second, how can business model approach help to identify opportunities and monetize security in future 5G? In the next two sub-chapters, we present two business model approaches that are suitable for ICT contexts, and, can help find answers to the above mentioned questions 
As the mobile telecommunications industry advanced, so did business model related discussions in the literature about extending organizational boundaries through vertical and horizontal integration in industries   Around this, Wirtz et al   offered four mutually exclusive business models classifying Internet based business models to be precise  According to Wirtz et al  , these business models present in Web 2 0 are related to connection, content, context, and commerce  Building further, Yrjölä et al   interpreted these business models as chronological layers, where “lower” layer business models are pre-requisites for the “higher” layer business models to exist 
Figure 1 The layered 4C ICT business models archetypes [Adapted from 35, 37, 38] 
The first layer is concerned with connection-related business model where a stakeholder provides connection services   Connection- related business models are relevant to connectivity for all sorts of devices and nodes through various communication channels, e g  PCs, smartphones, tablets, IoT devices, etc  The second layer is the business model focusing on monetizing content  At the content layer, all sorts of online content services (e g  mobile video streaming) are classified (i e  relevant, up-to-date and interesting) and are accessible conveniently for the end user  The content might be peer-to-peer/user-oriented contents (i e  exchange of personal content), and web browsing content (audio, video, text etc ) 
The third, context, layer concerns the ability to create and monetize user, content, equipment/user device and system profiles and turn (big) data into meaningful information and knowledge through systemic virtual contextualization  The fourth layer concerns commerce, the ability to monetize any or all of the connection, content, or context specific resources, actors or activities related to the ongoing communications  At this layer, we identify business, consumer and public/government types of communications   Thus, B2B (business-to-business), B2C and B2G communications as well as C2B, C2C and C2G or G2B, G2C and G2G communications may be monetized at this layer 
Casdesus-Masanell and Llanes’s  offers different software business models based on the openness of core software and extension software offering  According to the mixed source software business model approach, services can be: open source (open core- open extension), open core (open core- closed extension), open extensions (closed core- open extension) and proprietary (closed core- closed extension)  Although this model was offered specifically suiting the software industry, we argue that similar approach can be useful in other ICT contexts to scale up value offering by adjusting cooperation and industrial partnering model 
Figure 2 Mixed source business model approach [Adapted form 40] 
Depending on individual organization’s choice of mixed source options, it should eventually translate the value creation and value capture logic of the firm  Casadesus-Masanell & Llanes  argues that purely proprietary models results in higher captured value for organizations but lessens the scope of value creation for users  In contrast, purely open source models can deliver the maximum value to customers but reduce captured value  The key for organizations is to locate the best mix of openness-closeness for an optimum level of value capture and creation  From the business model elements collection, the mixed source approach explain key partners, key activities, key resources and technologies used  From a strategic perspective, this approach to business modeling helps an organization to find a way to scale up or down its business activities 
This study concerns business possibilities of IoT and mobile device management in the context of futures digital hospital  Since future is elusive and the speed of technology advancements has been rapid during recent decades, we consider the most suitable method for conducting this research to be through qualitative approach  Qualitative methods helps with flexibility and sensitivity to the context that is less explored, and it can facilitate understanding of how things work in a particularly complex setting   We adopt a single qualitative case study approach for this research to explain underlying business potential of a scantly explored industrial context within the existing literature  
Interview 1 Pediatric Surgery Administration, teaching, hospital development, technology and innovations Pediatric surgery patient journey, concept development of futures digital hospital, digitalization in healthcare 
Workshop 1 Network specialists, operator service provider, cybersecurity expert Business opportunity creation, new market exploration, security aspects Digitalization in healthcare, security business opportunities in health domain, business modeling 
Workshop 2 Operator service provider, cybersecurity expert Business opportunity creation, new market exploration, security aspects Digitalization in healthcare, security business opportunities in health domain, business modeling 
Table 1 summarizes the data used for this research  The empirical data used for this study was collected through two streams  First, in order to build the conceptual case of future digital hospitals, five interviews were conducted with pediatric surgery specialists, a nurse, and a general physician from two hospital districts in Finland  These semi-structured interviews were organized between April and May 2016  These interviews also focused on understanding the journey path in a pediatrics surgery case to clarify suitability of digitalization in the future  However, in the scope of this paper, we do not discuss the patient journey pediatrics surgery in detail  Second, we explored the cybersecurity and business perspectives in two workshops that were organized with telecommunications and cybersecurity experts from the industry  One of the workshops were organized in November 2016  The other workshop took place in August 2017  While the futures digital hospital context from a security perspective relates more towards hospital management and network functionality, the IoT devices and mobile devices perspective is closer to end user security needs  This paper focuses on the cyber security related business opportunities through IoT-MDM systems/services that broadly covers the above mentioned domains  Findings and analysis from the interviews and workshops are presented next 
In this section, we present our analysis in four steps  First, we briefly discuss the relevance of Internet business models (4C)  to the context of the study, addressing how this approach can help cybersecurity providers identify business opportunities in IoT-MDM  Second, we present a 4-quadrant matrix reflecting four different 5G security-provisioning scenarios, which are also relevant to IoT-MDM  Then, we connect each scenario back to business models by applying a mixed-source business model approach  and identify different possible business models for each kind of cybersecurity provider  Finally, we connect the overall discussion back to the case of future digital hospital 
The 4C business model archetypes helps decoding the boundaries between multiple business models operating either in the same verticals of the ecosystem or in the same horizontals  These archetypes can provide with a basis to classify and analyze business models of suppliers, competitors, and, at the same time, business models of customers  For this case, the 4C business model archetypes are mostly relevant for cybersecurity providers and IoT-MDM system/service providers as a tool to analyze customer business models and identify customer needs, the hospital management style  These customer needs can eventually be turned around as business opportunities  While IoT-MDM systems and services are not only concerned about cybersecurity, there is room for service providers to customize and tailor services based on customer needs  These service providers either can target customers from one layer (connection/content/context/commerce) across multiple industries, or, they can also target a specific industry (e g  healthcare) create customized services for all the layers in that industry 
To make sense of the overall cybersecurity provisioning for future 5G, we created the 4-quadrant matrix (Figure 3) by identifying major security drivers for new business creation in future 5G  These scenarios were created in workshops with telecommunication & cybersecurity experts from the industry  While these service provisioning scenarios are relevant to overall 5G, they are also significantly related to the case of IoT-MDM system/services 
Figure 3 Scenarios for 5G cybersecurity provisioning [Adapted from 35] 
As a result, we find four major drivers of security, which potentially will come with new business opportunities in the 5G era  Device driven security comprises distributed and D2D security techniques  Platform driven security will focus on centralized and D2D security techniques  Whereas, network infrastructure driven security should focus on centralized and infrastructure security methods  Lastly, location driven security should harness distributed and infrastructure security techniques 
In a quest to identify potential business entities operating in each of the quadrants relevant to device management, we recognize a ‘secure device manufacturer/provider’ focuses on device-driven security  Currently, multiple device manufacturers are developing devices where security features are built-in, regardless of which network or websites the users are accessing  This built-in security can be offered to multiple kinds of devices including smartphones, tablets, pcs, wearables, and even to IoT devices with communication and computation capability  These secure devices are built in a way that it will control access to potentially harmful networks, websites, and content; even without any commercially available security applications installed  However, when a customer enterprise (e g  the future digital hospital) is buying a fleet of hundreds of such secure devices, the question raises how to manage and configure all these devices from time-to-time  In such a case, even the secure devices will need IoT-MDM services for proper management and seamless upgrading when needed 
For the platform-driven security, we observe a ‘traditional IoT-MDM system provider’ can be a good example  In many cases, IoT-MDM system providers are selling there device management systems to enterprises directly  And, in other cases, they are selling the service through MNO’s bundled with connectivity and/or infrastructure  While as network infrastructure-driven security, a ‘mobile network operator/carrier’ or a ‘network infrastructure vendor’ can build own IoT-MDM system to offer their clients as well  And, finally, a location-specific micro operator can offer location-driven security  Micro operators offer mobile connectivity combined with specific, local services  The operation of a micro operator is spatially confined to either its premises or to a defined area of operation  As a part of the location-specific services, these micro operators can also offer IoT-MDM services for the users through outsourcing 
Further, we attempt to connect the aforementioned classification and examples of different players offering IoT-MDM services with the mixed source business model approach  Table 2 summarizes our understanding on how each kind of cybersecurity provider can open and mix the core value creation logic for end users  As mentioned previously, the mixed-source business model options are: open source (open core, open extensions), open core (open core, closed extensions), open extensions (closed core, open extensions), and proprietary (closed core, closed extensions)  In relation to these mixed-source options, we analyze the plausible options for each of the four distinct cybersecurity providers in the context of this study 
Secure device manufacturer/provider Secure devices Proprietary (own device, own IoT-MDM platform), Open extensions (own device, outsourced IoT-MDM service) 
From a secure device manufacturer perspective, device business can be considered as the core operation whereas IoT-MDM services would be extended solution  A secure device manufacturer/provider can have either a proprietary model or an open extensions model  In the proprietary model, the secure device manufacturer will offer their own devices alongwith their own IoT-MDM system/service  This is a viable case in a sense that customers who are purchasing the fleet of secure devices might prefer the IoT-MDM service from the same vendor, which is ideally less risk prone  Generally, when a customer organization buys a fleet of secure devices, they would expect the device management functions to be offered by the same service provider since integration of multiple systems/interfaces might increase vulnerability and reduce the customer’s confidence on the system  However, in practice, not all secure device manufacturer will specialize in automated device management solutions, and new R&D expenses to develop the IoT-MDM systems could result to be financially unfeasible  So, the second option for a secure device manufacturer is to have an open extensions model, where they will still offer the secure devices but outsource the IoT-MDM services to other vendors 
A micro operator’s core business relates to location specific network, while they are offering additional location specific services as extensions  Ideally, a micro operator’s business model in this case can be considered to be open source  At one hand micro operators are dependent on appropriate available spectrum resources on carriers and NIVs, and on the other hand, outsourcing the IoT-MDM services to other vendors seems more economically feasible than building own system  In contrast, a traditional IoT-MDM system/ service provider can have either a direct-open extensions business model or an indirect-open extensions business model  Unlike the other three archetypes in this discussion, IoT-MDM system/services are the core business for this kind of actor, as device and network related issues can be considered as extensions  An IoT-MDM system/service provider is characteristically dependent on connectivity providers, i e  MNOs, NIVs  In a direct-open extensions business model, while they are selling the IoT-MDM service directly to the end users, they are using the operator connectivity, but delivering their core value directly  Alternatively, in an indirect-open extensions business model, these IoT-MDM vendors can sell the services through MNOs/NIVs/device vendors to the end users 
Finally, a mobile network operator can either have an open extensions business model, or less likely a proprietary business model for IoT-MDM services  In an open extensions business model, operators will offer own connectivity to end users while outsourcing the IoT-MDM service to vendors  In very few cases though, operators might have their own IoT-MDM system on offering and thus they can sell both connectivity and IoT-MDM service as a proprietary bundle 
Looking back at the case of futures digital hospital, it is deemed to comprise various advanced technology-aided devices, let it be for clinical purposes or communications purpose that support healthcare  With the presence of devices like smartphones, tablets, wearables, connected TVs, VR touchpoints, AR touchpoints, robotic assistance for surgeries, thousands of smart IoT devices, makes the overall device network of the digital hospital complex and need automated centralized management  This centralized management of numerous IoT and mobile devices can be delivered through IoT-MDM system/services  Using the IoT-MDM system/service, the hospital organization can configure, secure, and time-to-time update their device network 
While procuring IoT-MDM services, the above analysis shows that futures digital hospitals can source it either directly from IoT-MDM service providers or through MNOs  Alternatively, if a digital hospital also plays the role of a micro operator, besides offering other location-specific services, the micro operator can also offer device management services by sourcing it from other vendors  In other cases, the future digital hospital can procure fleet of secure devices for the hospital from whom they can also source the device management service as a bundle  From the 4C business models perspective, the hospital organization seems to be a single organizational entity where different activities can be categorized in the 4C layers  Cybersecurity providers can specify and address such issues to highlight and customize their service offering for the future digital hospital 
In this article, we have looked at the futures digital hospital context from a device management perspective, and attempted to portray business model options based on how to create and capture economic value from cybersecurity business  Though in this research, we adopt a high-level stance on cybersecurity from a technical perspective, the overall discussions on business potential are relevant to issues like information security, communication security, storage security, security at vulnerable touchpoints in hospital context (end user interface layer, IoT nodes, system layer, network layer) 
Futures digital hospitals will be vulnerable to cybersecurity threats because of its data-dependency and digital-intensive device network  Thus, the business opportunity for cybersecurity providers in this case can be considered as the need for automated and centralized IoT-MDM service  To that end, this paper presents four distinct players who can provide such service to a critical infrastructure like a digital hospital  The mixed-source business model options further open up multiple alternatives that each type of vendor can consider while tailoring services for the future digital hospital   This paper also connects the 4C ICT business model archetypes to cybersecurity business context which can be used as an analytical tool to identify customer needs and scope for value creation  
Academically, this work contributes by filling up the void in discussing business models for cybersecurity as an industry  In addition, in the existing literature the hospital context has also been less discussed from a cybersecurity business perspective  From an industrial point of view, the business model options discussed in this research are timely and relevant to the market context and need  As mentioned, the mixed source business model options show how cybersecurity providers can extend their offering for different kinds of need for the hospital context based on their core businesses  This study can prove to be helpful for cybersecurity business entities and at the same time hospital managers  The scope of this paper explains business potential of cybersecurity vendors to an emerging industry from a higher level  In this paper, authors do not attempt to analyze the technical aspect of cyber security provisioning in healthcare context, however that is a forthcoming research possibility of this study  Also, since this research is based on a conceptual phenomenon, thus its empirical validation, both qualitatively and quantitatively is still yet to come, which can be considered as a limitation of the study 
All in all, we consider that applying a business perspective to IoT-MDM systems can solve many challenges of a modern mobile IT environment, not only in healthcare but also in other kinds of critical infrastructures   These IoT-MDM systems can be provided by various kind of vendors through a balanced and timely business model 
This study has been supported by the DIMECC Cyber Trust – Digital cyber security program 
 Lehto, I , and Ahokangas, P  (2017)  “Mobile Security Business Models for Critical Infrastructures – An Ecosystemic Approach”, in Proceedings of the 24th Nordic Academy of Management Conference 2017, Bodo, Norway 
 Loebbecke, C , and Picot, A  (2015)  Reflections on societal and business model transformation arising from digitization and big data analytics: A research agenda  The Journal of Strategic Information Systems, 24, 149–157 
 Himidan, S , and Kim, P  (2015)  The evolving identity, capacity, and capability of the future surgeon  In Seminars in pediatric surgery, 24, 145–149, WB Saunders 
 Beimborn, D , and Palitza, M  (2013)  “Enterprise app stores for mobile applications-development of a benefits framework”, in Proceedings of the Nineteenth Americas Conference on Information Systems, Chicago, Illinois 
 Ortbach, K , Brockmann, T , and Stieglitz, S  (2014)  “Drivers for the adoption of mobile device management in organizations”, in proceedings of the Twenty Second European Conference on Information Systems, Tel Aviv 2014 
 Wirtz, B  W , Schilke, O , and Ullrich, S  (2010)  Strategic development of business models: implications of the Web 2 0 for creating value on the internet  Long range planning, 43, 272–290 
 Afuah, A  (2004)  Business Models: A Strategic Management Approach  McGraw-Hill/Irwin 
 Alt, R , and Zimmermann, H  D  (2001)  Preface: introduction to special section–business models  Electronic markets, 11, 3–9 
 Chesbrough, H , and Rosenbloom, R  S  (2002)  The role of the business model in capturing value from innovation: evidence from Xerox Corporation’s technology spin-off companies  Industrial and corporate change, 11, 529–555 
 Shafer, S  M , Smith, H  J , and Linder, J  C  (2005)  The power of business models  Business horizons, 48, 199–207 
 Zott, C , and Amit, R  (2010)  Business model design: an activity system perspective  Long range planning, 43, 216–226 
 Xu, T , Wendt, J  B , and Potkonjak, M  (2014)  Security of IoT systems: Design challenges and opportunities  In Proceedings of the 2014 IEEE/ACM International Conference on Computer-Aided Design, 417–423  IEEE Press 
 Zhang, Z  K , Cho, M  C  Y , Wang, C  W , Hsu, C  W , Chen, C  K , and Shieh, S  (2014)  IoT security: ongoing challenges and research opportunities  In Service-Oriented Computing and Applications (SOCA), 2014 IEEE 7th International Conference, 230–234  IEEE 
 Farooq, M  U , Waseem, M , Khairi, A , and Mazhar, S  (2015)  A critical analysis on the security concerns of internet of things (IoT)  Int  J  Computer Appl  111 
 Backman, J , Väre, J , Främling, K , Madhikermi, M , and Nykänen, O  (2016)  IoT-based interoperability framework for asset and fleet management  In Emerging Technologies and Factory Automation (ETFA), 2016 IEEE 21st International Conference, 1–4  IEEE 
 Isotalo, L  (2017)  “5G Slicing as a Tool to Test User Equipment Against Advanced Persistent Threats”, in International Conference on Network and System Security, 595–603  Springer, Cham 
 Alcaraz, C , and Zeadally, S  (2015)  “Critical infrastructure protection: requirements and challenges for the 21st century”, in International Journal of Critical Infrastructure Protection, 8, 53–66 
 Sutherland, J , and van den Heuvel, W  J  A  M  (2006)  Towards an intelligent hospital environment: adaptive workflow in the OR of the future  In System Sciences, 2006  HICSS’06, in Proceedings of the 39th Annual Hawaii International Conference, 5, 100b–100b  IEEE 
 Haq, M  M  (2008)  U S  Patent No  7,412,396  Washington, DC: U S  Patent and Trademark Office 
 Spector, N  D , Cull, W , Daniels, S  R , Gilhooly, J , Hall, J , Horn, I , and Stanton, B  F  (2014)  Gender and generational influences on the pediatric workforce and practice  Pediatrics, 133, 1112–1121 
 Wirtz, B  W  (2011)  Business model management  Design–Instrumente–Erfolgsfaktoren von Geschäftsmodellen, 2 
 Gomes, J  F , Ahokangas, P , and Owusu, K  A  (2016)  Business modeling facilitated cyber preparedness  Int  J  Business & Cyber Security, 1, 54–67 
 Richardson, J  (2008)  The business model: an integrative framework for strategy execution  Strategic change, 17, 133–144 
 Morris, M , Schindehutte, M , and Allen, J  (2005)  The entrepreneur’s business model: toward a unified perspective  J  Business Res  58, 726–735 
 Zott, C , Amit, R , and Massa, L  (2011)  The business model: recent developments and future research  J  Manage  37, 1019–1042 
 Gomes, J F , Iivari, M , Ahokangas, P , Isotalo, L , Sahlin, B , and Melen, J  (Forthcoming, in press), Cyber security business models in 5G  Comprehensive guide to 5G security, Wiley Publishers, London 
 Ballon, P  (2007)  Business modelling revisited: the configuration of control and value  info, 9, 6–19 
 Yrjölä, S , Ahokangas, P , and Matinmikko, M  (2015)  Evaluation of recent spectrum sharing concepts from business model scalability point of view  In Dynamic Spectrum Access Networks (DySPAN, in 2015 IEEE International Symposiu, 241–250  IEEE 
 Ahokangas, P , Moqaddamerad, S , Matinmikko, M , Abouzeid, A , Atkova, I , Gomes, J  F , and Iivari, M  (2016)  Future micro operators business models in 5G  The Business & Management Review, 7, 143 
 Mitola, J , Guerci, J , Reed, J , Yao, Y  D , Chen, Y , Clancy, T , and Guo, Y  (2014)  Accelerating 5G QoE via public-private spectrum sharing  IEEE Communications Magazine, 52, 77–85 
 Casadesus-Masanell, R , and Llanes, G  (2011)  Mixed source  Manage  Sci  57, 1212–1230 
 Mason, J  (2002)  Qualitative researching  Sage 
 Miles, M  B , and Huberman, A  M  (1994)  Qualitative data analysis: An expanded sourcebook  Sage 
Julius Francis Gomes is pursuing his Ph D  in international business from the University of Oulu  He currently works at the Oulu Business School as a Doctoral Student to research the futuristic business models for digital intensive industries  His research focuses on using business models as a mean to look in to future industries  He is interested to research business ecosystems in different contexts like cyber security, healthcare, future’s network etc  with a business model perspective  He received his M Sc  (2015) in international business from the University of Oulu  Prior to that he acquired MBA (2011) specializing in managing information systems in business applications  Previously, he has also enjoyed about three years in a top tier bank in Bangladesh as a channel innovator 
Marika Iivari is a postdoctoral researcher at the Martti Ahtisaari Institute within Oulu Business School  She defended her doctoral dissertation on business models in ecosystemic contexts  She holds M Sc  in International Business from the Ulster University, Northern Ireland  Her research interests are in the areas of open innovation, business models and strategy in the context of innovation ecosystems and smart cities, digital and ICT business ecosystems  She has been involved in several research projects around 5G and the Internet of Things, most recently in the health care sector  She is also an active member of the Business Model Community, the Open Innovation Community and the Society for Collaborative Networks 
Petri Ahokangas received his M Sc  (1992) and D Sc  (1998) degrees from the University Vaasa, Finland  He is currently Adjunct Professor (International software entrepreneurship) and Senior research fellow at Martti Ahtisaari Institute, Oulu Business School, University of Oulu, Finland  His research interests are in how innovation and technological change affect international business creation, transformation, and strategies in highly technology – intensive or software – intensive business domains  He has over 100 publications in scientific journals, books, conference proceedings, and other reports  He is actively working in several ICT-focused research consortia leading the business-related research streams 
Lauri Isotalo has received his M Sc  from Helsinki University of Technology (currently Aalto University) in 1992  He has also a postgraduate Diploma in Business Administration  At first Lauri worked in Nokia Corp  in Mobile Technology & System Marketing unit specializing in Intelligent Networks  In 1992 he joined Elisa Corp  where he has held several managerial positions in value added services business, system and process security and mobile network development  Since 2005 Lauri has also led Elisa SME teams in various international collaboration projects and acquired a deep knowledge of the cyber security of legacy telecommunication networks, ip core, access networks, user terminals and modern virtualized data center IT platforms/cloud systems  From 2014 Lauri has headed SDN&NFV development in Elisa 
Riikka Niemelä, M Sc , MHSc , is a cross-disciplinary health-tech professional who obtained her degrees in Electrical Engineering (Master of Science) and Medical technology (Master of Health Sciences) from the University of Oulu, Finland  After R&D engineering, she worked as a research assistant to research the national eHealth development of the Finnish healthcare  Thereafter, she has been researching and promoting the adaptation of connected health technologies in surgical processes of a Nordic future hospital, resulting in scientific publications  Currently, she works as a project manager in Tuttlingen, Germany – the world center of Medical Technology – to generate R&D and innovation projects in cooperation with the ICT and LifeScience companies from Oulu, Finland  Her aim is to promote the digitalization and internationalization of the both high-tech regions and to generate innovations serving the increasing demands for healthcare 
Journal of ICT, Vol  5_1, 107–128 
Context 2: \section{Introduction}
Cognitive deficit of older adults is one of the biggest global public health challenges in elderly care  Approximately 5 2 million people of 65 and older are suffered with any form of cognitive impairments in United States in 2012 \cite{stat12}  Dementia is one of the major causes of the cognitive impairments which is more acute among 85 and older population (50\%) \cite{stat12}  However, the costs (financial and time) of health care and long-term care for individuals with Alzheimer's (special form of dementia) or other dementias are substantial  For example, during 2016, about 15 9 million family and friends in United States provided 18 2 billion hours of unpaid assistance to those with cognitive impairments which is a contribution to the nation valued at \$230 1 billion  One the other hand, total payments for all individuals with all form of cognitive impairments are estimated at \$259 billion  Total annual payments for health care, long-term care and hospice care for people with Alzheimer's or other dementias are projected to increase from \$259 billion in 2017 to more than \$1 1 trillion in 2050  Among the above costs, a significant amount are relevant to clinical and diagnostic tests \cite{stat17}  Although clinical and diagnostic tests have become more precise in identifying dementia, studies have shown that there is a high degree of underrecognition especially in early detection  However, there are many advantages to obtaining an early and accurate diagnosis when cognitive symptoms are first noticed as the root cause findings of impairment always lessen the progress of impairment status and sometimes symptoms can be reversible and cured 

With the proliferation of emerging ubiquitous computing technologies, many mobile and wearable devices have been available to capture continuous functional and physiological behavior of older adults  Wearable sensors are now capable of estimating number of steps being taken, physical activity levels, sleep patterns and physiological outcomes (heart rate, skin conductance) of older adults \cite{sano15}  Ambient sensors also help capture the movement patterns of objects and humans for activity and behavior recognition \cite{dawadi14,dawadi15}  Researchers also proved the existence of correlations between cognitive impairment and everyday task performance \cite{dawadi14, akl15,alam16} as well as physiological symptoms \cite{alam16,sano15}  Although current studies showed some successes in IoT-assisted cognitive health assessment in different domains individually, there are several existing challenges in developing and validating a fully automated multi-modal assessment model 

\begin{enumerate}
\item \emph{Real-time IoT System}: A real-time IoT system must include a continuous and fault tolerant data streaming capability among central hub, wearable sensors and ambient sensors regardless of network communication protocol (WiFi, Ethernet, Bluetooth etc ) which are not available in existing researches 
\item \emph{Multi-modal Context Fusion}: Though several offline clinically validated cognitive health assessment tools exist \cite{wai03, starling99, krapp07, yesavage82, zung71}, there is no universally accepted method for IoT-assisted automatic cognitive health assessment in smart home environment that can fuse multi-modal sensor contexts altogether  For example, some researchers showed ambient sensors based Activities of Daily Livigin (ADLs) sequence pattern can signify the cognitive health status of older adults \cite{akl15, dawadi15}  Researchers also showed wearable Electrodermal Activity pattern analysis may carry the significance of cognitive status \cite{sano15}  However, for validation of IoT based cognitive health assessment, self-reported surveys, clinical diagnosis and observation based tools are used individually by prior researchers \cite{akl15, dawadi15, sano15, alam16} 
\end{enumerate}

Regarding aforementioned challenges for the automation of cognitive health assessment, \emph{AutoCogniSys} considers (i) reproducibility of our model in any smart home system consists of ambient motion sensors, wearable accelerometer (ACC) sensors, wearable Electrodermal Activity (EDA) and Photoplethysmography (PPG) sensors individually or combined streams; (ii) context awareness based on ambient motion sensors and wearable ACC sensors in any types of activities such as hand gestural, postural and complex ADLs; and (iii) high accuracy, i e , a recall rate of over 90\% with less than 5\% false positive rate  More specifically, \emph{AutoCogniSys} extends our existing work \cite{alam16} in three dimensions,

\emph{(1) True Automation:} We first investigate the correlations of cognitive impairment with human activities and stress where we manually labeled activities, extract the corresponding physiological sensor (EDA and PPG) features of each activity, and use statistical method to find correlations  Then, we propose automatic complex activity recognition based on a Hierarchical Dynamic Bayesian Network (HDBN) model, fine-grained extraction of physiological sensor features and finally machine learning classification of cognitive impairment 

\emph{(2) Noises Elimination:} We define different types of noises on ACC, EDA and PPG sensors, propose extensive signal processing techniques to remove noises and show significant improvement can be achieved in cognitive impairment classification 

\emph{(3) Implementation and Evaluation:} Finally, we design and implement IoT system and analytic methods and minimize the human involvement to automate our proposed cognitive health assessment approach by considering effective smart home sensor customization and deployment, data collection, screening, cleaning and filtering, feature computation, normalization and classification, and activity  model training 

\textbf{Research Questions:} \emph{AutoCogniSys} consequently tackles the following key research questions 

$\bullet$ Can we detect simultaneously the periodic rhythms of  both hand gestures and postural activities from wrist-worn ACC sensor signal for diverse population (population with same activity but diverse ways such as walking with walker, stretcher or normally)? If so, how can we incorporate the hand gesture, posture and ambient sensor data streams to help improve the ADLs recognition models?

$\bullet$ How can we exploit and relate the micro-activity features into noise free physiological sensor signals processing to automate cognitive health assessment process? What are the critical roles of clinical survey and technology guided assessment methodologies and their inter-relationships for automating the different intermediate steps of cognitive health assessment process?

To tackle these, we make the following \textbf{key contributions}:

$\bullet$ We employ an extensive signal deconvolution technique that in conjunction with machine learning technique helps facilitate a wrist-worn ACC-based multi-label (hand gestural and postural) activity recognition for diverse population  We then leverage multi-label context sets with ambient and object sensor signals for complex activity recognition based on HDBN model 

$\bullet$ We propose a novel collaborative filter for EDA signal processing by postulating signal as a mixture of three components: \emph{tonic phase, phasic phase} and \emph{motion artifacts}, and employ convex optimization technique for filtering out the motion artifacts  We also propose a novel PPG signal processing technique to filter out the inherent motion artifacts and noises using improved Periodic Moving Average Filtering (PMAF) technique 

$\bullet$  We design and prototype an IoT system consisting of multiple devices (wearable wrist band, IP camera, object and ambient sensors) connected with central hub via WiFi, Ethernet and Bluetooth communication protocols  We collected data from 22 older adults living in a continuing care retirement community center in a very natural setting (IRB \#HP-00064387) 

$\bullet$ Finally, we employ statistical and machine learning techniques to jointly correlate the activity performance metrics and stress (EDA and PPG) features that helps achieve max  93\% of cognitive impairment status detection accuracy  We evaluate \emph{AutoCogniSys} on 5 clinically validated offline assessment tools as ground truth 
\section{Related Works}
\emph{AutoCogniSys} builds on previous works on wearable devices based low-level (postural and hand gestural) activity recognition and their integration with ambient sensors to recognize complex ADLs, the underlying signal processing and applications on cognitive health assessment automation 
\subsection{Wearable Sensor Signal Processing}
Wearable sensors can be two types: physical and physiological  Physical sensors (accelerometer, gyroscope etc ) signal values change over the movements of the sensor devices  Physiological sensors change over physiological condition of body such as EDA changes over stress and PPG changes over heart rate  However, physical movements also impose noises on physiological sensor signals which is called \emph{motion artifacts} 
\subsubsection{Physiological Signal Processing}
A continuous and descrete decomposition of EDA, and time and frequency domain analytics of PPG signal have been investigated before to extract relevant physiological features which were contaminated with noises and motion artifacts \cite{alam16}  \cite{setz10} denoised and classified EDA from cognitive load and stress with accuracy higher than 80\%  Though motion artifacts removal techniques such as exponential smoothing \cite{hern11} and low-pass filters \cite{poh10, hernandez14} provide significant improvement in filtering EDA signals, wavelet transforms offer more sophisticated refinement for any kind of physiological sensors such as electroencephalogram \cite{krish06, zikov02}, electrocardiogram \cite{erc06,alfa08}, and PPG \cite{lee03}  \cite{chen15} proposed a stationary wavelet transform (SWT) based motion artifacts removal technique  `cvxEDA' proposed  a convex optimization technique considering EDA as a mixture of white gaussian noise, tonic and phasic components where white gaussian noise includes motion artifacts and external noises \cite{greco16}  \emph{AutoCogniSys} intelligently combines SWT and `cvxEDA' together to remove noises and motion artifacts from EDA signal  On the other hand, it is more difficult to remove motion artifacts from PPG signal due to its periodicity of nature \cite{wang13}  Researchers proposed different methods such as frequency analytics \cite{garde13,wang13}, statistical analytics \cite{peng14} and digital filter \cite{lee10} to reduce noises and motion artifacts from PPG  \emph{AutoCogniSys} used Periodic Moving Average Filter (PMAF) in this regard \cite{lee07} 
\subsubsection{Physical Sensor Signal Processing}
ACC based hand gesture recognition has been explored by several researchers in past such as discrete hidden markov model \cite{liu10}, artificial neural network \cite{arce11}, weighted naive bayes and dynamic time warping \cite{mace13}  Akl et  al  proposed 18 gesture dictionary based Support Vector Machine (SVM) classifier \cite{akl11}  Wrist-worn ACC based postural activity recognition approach has been proposed using Decision Tree, Random Forest, Support Vector Machines, K-Nearest Neighbors, Naive Bayes and deep neural networks \cite{gj14, wang16}, the accuracy stagnates at 85\% using SVM method \cite{martin16}  However, neither of past works proposed any technique that can provide single body worn ACC sensor-based multiple body contexts recognition nor works efficiently for diverse posture say walking normally, with walker, with double walker or wheel chair  Our proposed 8-hand gesture recognition technique assisted sparse-deconvolution method improves classification performances on both normal and diverse postures  However, we incorporated hand gestures and postures in conjunction with ambient sensors into single-inhabitant HDBN model \cite{alam16b} that provides significant improvement in complex activity recognition 
\subsection{Cognitive Health Assessment}
Smart home environment has been used for providing automated health monitoring and assessment in the ageing population before \cite{dawadi14, gong15, akl15, dawadi15}  `SmartFABER' proposed a non-intrusive sensor network based continuous smart home environmental sensor data acquisition and a novel hybrid statistical and knowledge-based technique to analyz the data to estimate behavioral anomalies for early detection of mild-cognitively impairment \cite{riboni16}  \cite{skubic15} presented an example of unobtrusive, continuous monitoring system for the purpose of assessing early health changes to alert caregivers about the potential signs of health hazards  Though, prior researches proposed a sequence of ambient motion sensor streams as complex activity components in activity based health assessment \cite{dawadi14, gong15, akl15, dawadi15}, we consider inclusion of an wearable wrist-band with in-built ACC sensor to detect hand gesture and posture, augmenting with the ambient sensor readings to help recognize complex activities as well as cognitive health assessment of older adults  Additionally, we propose intelligent use of physiological features of skin through different physiological sensor signals (EDA, PPG) processing in daily activity tasks and incorporate context-awareness for automation of cognitive health assessment that have not been explored before 
\begin{figure}[!htb]
\begin{center}
   \epsfig{file=flowchart pdf,height=1 6in, width=3 5in}
\caption{Overall flow of \emph{AutoCogniSys} pipeline }
   \label{fig:overview}
\end{center}
\end{figure}
\section{Overall Architecture}
We first investigate existing IoT-based cognitive health care frameworks that covers every aspects of wearable (physical, physiological) and ambient (passive infrared and object) sensor signals computing  \emph{AutoCogniSys} is comprised of three component modules: (i)~sensing, (ii)~processing, and (iii)~analysis  The `sensing' module consists of clinical assessment tools (surveys, observation and clinical backgrounds) and sensing signals (ambient and wearable sensors)  `Sensor processing' module is comprised of three sub-modules: a)~clinical assessment feature extraction from assessment tools; b)~ambient sensor feature extraction; and c)~wearable sensor processing (noise removal, segmentation, feature extraction, classification etc )  `Analysis' module is comprised of machine learning and statistical analytics-based score prediction of cognitive impairment  Automation of each module's functionality and inter-intra modular transactions without human interference can be called {\it true automation} of cognitive health assessment  Fig ~\ref{fig:overview} shows the overall flow of \emph{AutoCogniSys} which is discussed in details in the following sections 
\subsection{Demographic Ground Truth Data Collection}
Currently in standard clinical practice and research, the most accurate evaluations of cognitive health assessment are one-to-one observation and supervision tasks/questionnaires for monitoring an individual's functional abilities and behavior \cite{resnick15}  In the first stage of this pilot study, we have investigated current literatures and carefully chosen the clinically proven functional and behavioral health assessment survey tools \cite{resnick15}  On the other hand, to cross check with the survey based evaluations, we have also chosen clinically justified observation based behavioral assessment methods  First, following the resident consent, our clinical research evaluator collects demographic and descriptive data (age, gender, race, ethnicity, marital status, education and medical commodities)  She has performed two types of clinical assessments: (1) \emph{Observation based} where the resident's cognition is assessed using the Saint Louis University Mental Status (SLUMS) scale \cite{wai03}  (2) \emph{Survey based} where five widely used and clinically well validated surveys are taken into account: (a) \emph{Yale Physical Activity Survey} \cite{starling99}; (b) \emph{Lawton Instrumental Activities of Daily Living}; (c) \emph{Barthel Index of Activities of Daily Living} \cite{krapp07}; (d) \emph{Geriatric Depression Rating scale} \cite{yesavage82}; and (e) \emph{Zung Self-Rating Anxiety scale} \cite{zung71} 
\subsection{Smart Environment Creation}
For an ideal IoT-based system, instrumenting and deploying it at each participant's natural living environment warrants for assembling a flexible set of hardware and software interfaces to ease the system configuration, setup, and network discovery processes  The sensor system placed in the residences of volunteers needs to meet several specific physiological signals and activity monitoring needs  However, we must confirm that the devices are reliable with potential for re-deployment as well as appear unintimidating to the participants  Inspired by the above requirements, we developed a real testbed IoT system, {\it SenseBox}, by customizing Cloud Engine PogoPlug Mobile base station firmware to integrate with WiFi (connect ambient and object sensors) and Bluetooth (connect wristband) protocol  The smart home components are as follows: (i) PogoPlug base server with a continuous power supply, (ii) 3 binary Passive Infrared sensors in three different rooms (kitchen, livingroom and bedroom) to capture room level occupancy, (iii) 7 binary object sensors attached with closet door, entry door, telephone, broom, laundry basket, trash can and trash box, (iv) three IP cameras in the appropriate positions to collect the ground truth data and (v) an Empatica E4 \cite{empatica} wrist-band (integrated sensors: PPG at 64 Hz, EDA at 4 Hz, Body temperature at 1 Hz and a triaxial ACC at 32 Hz) on the participant's dominating hand 
\section{Activity Recognition}
We aim to detect single wrist-worn ACC sensor based hand gesture and postural activities and insert these into an HDBN graphical model in conjunction with ambient and object sensor values for complex activity recognition  We consider the recognition problem asan activity tupple of $\langle gesture,posture,ambient,object \rangle$  Though, Alam et  al  provides significant performance improvement for single wrist-worn ACC sensor aided 18-hand gesture based postural activity recognition in lab environment \cite{alam17}, it faces some practical challenges in real-time smart environment with older adults due to the diversity of their postures  For example, some older adults use walker, double walking sticks or wheel chair for walking in which cases collecting 18 hand gestures and corresponding postural activities for training requires endless efforts and carefulness  To reduce the complexity of ground truth labeling and later state space explosion for graphical model (HDBN), we propose to use rotational normalization method that can merge some hand-gestures subject to directional differences and forms an 8-hand gesture model  However, our proposed Feature Weight Naive Bayes (FWNB) classifier adds significant improvement on Alam et  al  proposed sparse-deconvolution method as well as recognition in diverse postural environment 
\begin{figure}[!htb]
\begin{center}
   \epsfig{file=hand_gestures pdf,height=0 5in, width=3in}
   \vspace{- 2in}
\caption{8 hand gesture dictionary with direction}
   \label{fig:hand_gestures}
   \vspace{- 2in}
\end{center}
\end{figure}
\subsection{Hand Gesture Recognition}
\label{sec:hand_gesture}
\emph{AutoCogniSys} proposes an 8-gesture dictionary (as shown in Fig  \ref{fig:hand_gestures}) and a Feature Weighted Naive Bayesian (FWNB) framework for building, modeling and recognizing hand gestures  The method comprises of the following steps: (i) \emph{Preprocessing:} wrist-worn ACC sensor provided 3-axis data are passed through 0 4Hz low-pass filter to remove the data drift  (ii) \emph{Rotation normalization:} Normalizing the rotation of hand gestures provides greater accuracy and allows for more realistic, orientation-independent motion  At first, we find the best fit plane of the acceleration vectors thus if the motion lies in a single plane, then the acceleration vectors of a closed shape should on average lie in that main plane  Then, we take all acceleration segments between points of inflection to form one single vector called reference vector that provides us the general direction of user's motion  After that, each vector is normalized relative to the reference vector  This normalization helps remove a lot of hand gestures from prior considered 18 hand gestures resulting a reduced dictionary of 8 gestures  (iii) \emph{Feature Weighted Naive Bayesian model:} Naive Bayes classifier is light-weight and efficient technique for hand gesture recognition  We extract 12 ACC features \cite{alam17} and calculate weight for each feature type based on the similarity of feature measures of the trained gestures (0$<$weight$<$1)  While recognizing gestures, the proximity of each feature measure to the average trained feature measure of each gesture type is calculated by a normal distribution  Then, the proximity value is multiplied by the feature weight that was calculated in the training phase   All of these multiplied values are added together and the system predicts the gesture type with the greatest value as the user gesture  In the learning data points, there should be static postural activities (such as sitting, lying etc ) to avoid unexpected noises over wrist-worn ACC sensors  In the final hand gesture dictionary, we save the reference vector as our signal dictionary 
\subsection{Postural Activity Recognition}
In normal lab environment, wrist-worn ACC sensor signal is a mixture (convolution) of actual hand gesture and postural activity relevant signals \cite{alam17}  \emph{AutoCogniSys} improves the idea by reducing the number of hand gestures and postural activities to 8 (as shown in Fig \ref{fig:hand_gestures}) using rotation normalization and 4 (walking, sitting, standing and lying)  Then, we use sparse-deconvolution method (with 31\% signal reconstruction error) to get Approximately Sparse Factor  The summary of the entire process is stated bellow:

{\it Building Deconvolution Method:} We first consider the wrist-worn ACC sensor signals (3-axis values) as a convolution of hand gesture and postural activity effects and build a deconvolution framework  The deconvolution framework takes a known signal (hand gesture effects) and a equalizer parameter ($\lambda$) as input and provides an Approximately Sparse Factor signal (postural activity effects) as output  For 3-axis ACC signals, we need to learn associated 3 equalizer parameters for each hand gesture  Moreover, each equalizer parameter is involved with 4 postural activities that results a total 96 ($8\times 3\times 4$) equalizer parameters to learn  

{\it Learning Classification Model:} We use the Approximately Sparse Factor signal to extract 12 statistical features and SVM with sequential machine optimization (SMO) \cite{cao06} for postural activity recognition 

{\it Prediction Model:} After recognizing the hand gestures following the method explained in Sec ~\ref{sec:hand_gesture}, we take the corresponding reference vector as known signal and extract the Approximately Sparse Factor signals incorporating corresponding 3 equalizer parameters ($\lambda$) for the sparse-deconvolution method  Then, we apply feature extraction and prior learned SMO based SVM classifier \cite{cao06} to classify final postural activity  Fig ~\ref{fig:deconvolution} illustrates a single axis example of the deconvolution 

\begin{figure}[!htb]
\begin{center}

   \epsfig{file=deconvolution pdf,height=1 6in, width=3in}
   \vspace{- 15in}
\caption{Sample deconvolution example of X-axis  The raw x-axis of accelerometer signal, reference vector of the sample gesture and the extracted corresponding ASF signal of walking }
   \label{fig:deconvolution}
\end{center}
\vspace{- 15in}
\end{figure}

\subsection{Complex Activity Recognition}
We build a HDBN based complex activity recognition framework for single inhabitant scenario smart home environment \cite{alam16b} taking the advantage of detected hand gestural and postural activities along with the ambient and object sensor streams  At first, we obtain instant hand gestural and postural activities from our above proposed models, and additionally motion sensor and object sensor readings from our IoT-system for every time instant generating a 4-hierarchy of HDBN model  Considering the context set $\langle gestural, postural, ambient,object\rangle$ as a hierarchical activity structure (extending two 2-hierarchical HDBN \cite{alam16b}), we build complex activity recognition model for single inhabitant scenario  Finally, we infer the most-likely sequence of complex activities (and their time boundaries), utilizing the well-known Expectation Maximization (EM) algorithm \cite{dempster77} for training and the Viterbi algorithm \cite{forney73} for run-time inference 
\section{Automatic Activity Features Estimation}
The effects of cognitive ability on daily activity performance have been studied before \cite{dawadi14,akl15}  They experimentally and clinically validated that cognitive impairment highly reduces the daily activity performances and this activity performance can be computed as an indicator of cognitive ability status of older adults  The standard activity features refer to completeness of task (TC), sequential task ability (SEQ), interruption avoidance capabilities (INT) etc  In current behavioral science literature, the above activity features carry specific definition based on the sub-tasks involved with a complex activity \cite{dawadi14,akl15}  Completeness of task refers to how many sub-tasks are missed by the participants  Sequential task ability refers to how many sequences of sub-tasks are missed referring the gerontologist defined standard sequences of the sub-task for the particular complex activity  Interruption avoidance capability refers to how many times the participants stop or interleave while doing any sub-task  The final goal of activity features estimation is to provide overall task score  The task score is proportional to the functional ability of participants in performance daily activities  Our behavioral scientist team, comprises with Nursing professor, gerontologist and retirement community caregivers, carefully discus, optimize and choose 87 sub-tasks in total for 13 complex activities 

Each of the sub-task comprises with sequential occurrences of hand gesture and postural activities  However, no researchers ever considered hand gesture for activity features estimation due to complexity of multi-modal wearable and ambient sensors synchronization and multi-label activity classification \cite{dawadi14,akl15}  \emph{AutoCogniSys} exploited single wrist-worn sensor based hand gesture and postural activity recognition, and proposed an activity features (TC, SEQ and INT) estimation method including these two parameters in conjunction with object and ambient sensor features that provide significant improvement of cognitive health assessment of older adults 
\subsection{Machine Learning Based Complex Activity Features Estimation}
In current cognitive health assessment literature, complex activity features can be defined as $\langle TC,SEQ,INT,TS\rangle$  We used supervised method to estimate TC, SEQ and INT, and unsupervised method to estimate TS  We first, formulate the automated scoring as a supervised  machine learning problem in which machine learning algorithms learn a function that maps $\langle${\it hand gesture, posture, object, ambient sensor}$\rangle$ feature set to the direct observation scores  We use bagging ensemble method to learn the mapping function and SMO based SVM \cite{cao06} as base classifier  The learner averages by boostrapping individual numeric predictions to combine the base classifier predictions and generates an output for each data point that corresponds to the highest-probability label  We train three classifiers considering observation as ground truth for TC, SEQ and INT scores and test on the testing dataset  We derive unsupervised scores using dimensionality reduction technique for each feature set  First, we take all features of each activity, apply optimal discriminant analysis technique as a dimensionality reduction process \cite{zhang09} and reduce the feature sets into single dimensional value which represents the automated task completeness scores of the particular user activity  A min-max normalization is applied that provides us a uniform range of the variables using $
z_i=\frac{x_i-min(x)}{max(x)-min(x)}$ equation where $x=\{x1,\ldots,x_n\}$ and $z_i$ is $i^{th}$ normalized data  The final single dimensional score represents machine learning based TS score 
\section{Physiological Sensor Signals Processing}
The autonomic nervous system (ANS) restrains the body's physiological activities including the heart rate, skin gland secretion, blood pressure, and respiration  The ANS is divided into sympathetic (SNS) and parasympathetic (PNS) branches  While SNS actuates the body's resources for action under arousal conditions, PNS attenuates the body to help regain the steady state  Mental arousal (say stress, anxiety etc ) activates the sweat gland causing the increment and reduction of  Skin Conductance on SNS and PNS physiological conditions respectively  However, Instant Heart Rate also has similar effect on SNS and PNS physiological condtions i e , a higher value of heart rate is the effect of SNS and lower value is the outcome of PNS  EDA and PPG sensors are widely used to estimate the instant value of skin conductance and heart rate respectively \cite{alam16} 
\subsection{EDA Sensor Signal Processing}
EDA is the property of the human body that causes continuous variation in the electrical characteristics of the skin which varies with the state of sweat glands in the skin  There are three types of arousal: \emph{cognitive, affective and physical}  \emph{Cognitive} arousal occurs when a person tries to solve any problem using her cognitive ability  \emph{Affective} arousal occurs when a person is worried, frightened or angry either doing daily activities or in resting position  On the other hand, \emph{physical} arousal is related to the brain command to move bodily parts which is imposed on the total arousal as an artifact, called \emph{motion artifact}  However, there are always some noises due to the weather conditions (temperature, humidity etc ) and device motion  This \emph{motion artifact} can be the prime cause of signal contamination of physiological outcomes while performing daily activities which must be removed  \emph{AutoCogniSys} proposes an EDA sensor signal processing method consists of three steps: (i) noise and motion artifacts removal, (ii) separation of tonic component and phasic component (explained later) from contamination free EDA signal and (iii) feature extraction on the response window 
\subsubsection{Motion Artifacts Removal}
There are many types of motion artifacts but the unsual steep rise is the mostly occured ones associated with EDA signal while performing daily activities \cite{edel67}  We use well-known steep rising noises reduction technique, SWT \cite{chen15}  We first consider EDA signal as a mixture of a slow variant tonic and fast variant phasic component, i e , SWT coefficient is modeled as a mixture of two Gaussian components, phasic (close to zero valued signal) and tonic (high rising signal)  After expanding EDA signal into multiple levels of scaling and wavelet coefficients, we choose adaptively a threshold limit at each level based on the statistical estimation of the wavelet coefficients' distribution, and employ that on the wavelet coefficients of all levels  Finally, an inverse wavelet transform technique is applied to the thresholded wavelet coefficients to obtain the artifacts free EDA signal  Fig~ \ref{fig:eda_artifact_removal} shows a sample of raw and motion artifacts free EDA signal 

\begin{figure}[!htb]
\begin{center}
\vspace{- 1in}
   \epsfig{file=eda_signal_artifact pdf,height=1 6in, width=3 5in}
\caption{Dashed line represents noisy EDA signal and solid red line represents \emph{AutoCogniSys} proposed motion artifact free EDA signal}
   \label{fig:eda_artifact_removal}
\end{center}
\end{figure}
\subsubsection{Convex Optimization Technique to EDA Deconvolution}
After the motion artifact removal, we consider EDA as the sum of three components for $N$ sample: a slow tonic driver ($t$), fast (compact, bursty) non-negative sparse phasic driver ($r$) and a reminder error term ($\epsilon_r$) 
\begin{equation}
\label{eq:eda_signal}
y = t + r + \epsilon_r
\end{equation}
This additive error $\epsilon_r$ is a White Gaussian Noise  The central problem associated with the deconvolution method is to get tonic $t$ component from the above equation  \cite{greco16} showed that EDA signal deconvolution (separation of tonic, phasic and noise terms from EDA signal) is a quadratic optimization problem and defined tonic component as follows:
\begin{equation}
\label{eq:tonic}
t = Bl + Cd,
\end{equation}
where $B$ is a tall matrix whose columns are cubic $B$-spline basis functions, $l$ is the vector of spline coefficients, $C$ is a $N\times 2$ matrix, $d$ is a $2\times 1$ vector with the offset and slope coefficients for the linear trend  The above equation is subject to the following optimization problem,
\begin{eqnarray}
minimize \frac{1}{2} {||Mq + Bl + Cd- y||}^2_2 +\alpha {||Aq||}_1 + \frac{\lambda}{2} {||l||}^2_2\\
subject\;to\; Aq \geq 0\nonumber
\end{eqnarray}
where $M$ and $A$ are tridiagonal matrices and $q$ is an auxiliary variable  After solving the above equation, we can get the optimal values for $\{q,l,d\}$ that can be used to obtain tonic component from the equation~\ref{eq:tonic}  The reminder of the equation~\ref{eq:eda_signal} ($r+\epsilon_r$) is considered as a mixture of White Gaussian Noise ($\epsilon_r$) and a fast variant phasic component ($r$)  We employ butterworth low-pass filter (5Hz) and hanning smoothing with window size 4 (optimal) to remove $\epsilon_r$ from phasic component ($r$) 
\subsection{PPG Signal Processing}
PPG is used mainly for measuring the oxygen saturation in the blood and blood volume changes in skin  An ideal PPG signal processing must contain the following steps: noise and motion artifacts removal, heart rate detection, heart rate variability estimation and feature extraction 
\subsubsection{PPG Signal Noise and Motion Artifacts Removal}
Similar to EDA signal, PPG signal is also contaminated with motion artifacts and noises  However, unlike EDA signal, PPG produce quasiperiodicity in a time series spectrum \cite{mete30}  We use Periodic Moving Average Filter (PMAF) to remove motion artifacts and noises \cite{lee07}  We first segment the PPG signal on periodic boundaries and then average the $m^{th}$ samples of each period  After filtering the input PPG signal with a 5-Hz $8^{th}$-order Butterworth low-pass filter, we estimate the maximum and minimum value of each period  The mean of each period are obtained from the maximum and minimum values applying the zero crossing method  These points of the means help determine the boundaries of each period  Then, interpolation or decimation is performed to ensure that each period had the same number of samples \cite{lee07}  
\subsubsection{Heart Rate and Heart Rate Variability Estimation}
We first apply PMAF on PPG signal to remove noises and motion artifacts, refine PPG by smoothing the signal using 1-dimensional Gaussian Filter and Convolution, calculate first derivative of the convoluted signal and finally find the differences between two consecutive peak values which is called HRV \cite{sel08}  The occurrences of total peak values (R-peak or beat) in each minute is called Heart Rate (HR) with an unit of Beat Per Minute  The signal value property of HRV and HR are inversely proportional which means the mental arousal that increases HR should decrease HRV in the time segment window  Fig~\ref{fig:ppg_artifact_removal} shows a sample of the noisy and filtered PPG signal and their corresponding Instant Heart Rate 
\begin{figure}[!htb]
\vspace{- 1in}
\begin{center}
   \epsfig{file=ppg_artifact_removal pdf,height=1 4in, width=3 5in}
   \vspace{- 15in}
\caption{Top figure illustrates the noisy signal (dotted line) and filtered signal from PPG sensor based on our filtering method  Bottom figure illustrates instant heart rate calculated from noisy signal (dotted line) and filtered signal}
   \label{fig:ppg_artifact_removal}
\end{center}
\vspace{- 15in}
\end{figure}
\subsection{Physiological Sensor Signal Feature Extraction}
Using the above mentioned methods, we removed the noises and motion artifacts from EDA and PPG signals and generated two time series signal from EDA (tonic and phasic components) and one time series signal from PPG (HRV)  Then, we segment each of the time series signal based on our prior detected complex activities such that each response window starts and ends with the starting and ending points of each complex activity  We extract 7 statistical time-series features for EDA (as shown in Table~\ref{tab:eda_features}) and 8 features for HRV (Table~\ref{tab:hrv_features}) within the response window) 

\begin{table}[!t]
\begin{center}

\renewcommand{\arraystretch}{1}
\caption{EDA Features Within The Response Window}
\begin{scriptsize}


\label{tab:eda_features}
\begin{tabular}{|c|l|}
\hline
\bfseries Features& \bfseries Description\\
\hline
nSCR & Number of SCRs within response window (wrw)\\
\hline
Latency & Response latency of first significant SCR wrw\\
\hline
AmpSum & Sum of SCR-amplitudes of significant SCRs wrw\\
\hline
SCR & Average phasic driver wrw\\
\hline
ISCR & Area (i e  time integral) of phasic driver wrw\\
\hline
PhasicMax & Maximum value of phasic activity wrw\\
\hline
Tonic & Mean tonic activity wrw\\
\hline
\end{tabular}
\end{scriptsize}
\end{center}
\end{table}



\begin{table}[!t]
  \begin{center}
\renewcommand{\arraystretch}{1}
\vspace{- 3in}
\caption{Heart Rate Variability features}
\label{tab:hrv_features}
\begin{scriptsize}
\begin{tabular}{|c|l|}

\hline
\bfseries Feature& \bfseries Description\\
\hline
$\overline{RR}$&Mean RR intervals\\
\hline
SDNN&Standard deviation of RR intervals\\
\hline
SDSD&Std of successive RR interval differences\\
\hline
RMSSD&Root mean square of successive differences\\
\hline
NN50&\#successive intervals differing more than 50 ms\\
\hline
pNN50&relative amount of NN50\\
\hline
HRVTI&Total number of RR intervals/height of the histogram\\
\hline
TINN&Width of RR histogram through triangular interpolation\\
\hline
\end{tabular}
\end{scriptsize}
  \end{center}
\end{table}
\section{Experimental Evaluation}
In this section, we explain our data collection, available benchmark dataset, baseline methods and evaluation 
\subsection{Datasets and Baseline Methods}
We validate and compare \emph{AutoCogniSys} with baseline methods on both publicly available and our collected datasets 
\subsubsection{RCC Dataset: Collection and Ground Truth Annotation}
For collecting Retirement Community Center Dataset (RCC Dataset), we recruited 22 participants (19 females and 3 males) with age range from 77-93 (mean 85 5, std 3 92) in a continuing care retirement community with the appropriate institutional IRB approval and signed consent  The gender diversity in the recruited participants reflects the gender distribution (85\% female and 15\% male) in the retirement community facility  A trained gerontology graduate student evaluator completes surveys with participants to fill out the surveys  Participants are given a wrist band to wear on their dominant hand, and concurrently another trained IT graduate student have the IoT system setup in participants' own living environment (setup time 15-30 minutes)  The participants are instructed to perform 13 \emph{complex ADLs}  Another project member remotely monitors the sensor readings, videos and system failure status  The entire session lasts from 2-4 hours of time depending on participants' physical and cognitive ability 

We follow the standard protocol to annotate demographics and activities mentioned in the IRB  Two graduate students are engaged to annotate activities (postural, gestural and complex activity) whereas the observed activity performances are computed by the evaluator  Two more graduate students are engaged to validate the annotations on the videos  In overall, we are able to annotate 13 complex activities (total 291 samples) labeling for each participant; 8 hand gestures (total 43561 samples) and 4 postural activities (total 43561 samples) labeling  Annotation of postural and complex activities outcomes no difficulties from recorded videos  However, annotation of hand-gestures is extremely difficult in our scenario  We used video based hand tracker that can track and sketch wrist movements from a video episode \cite{hugo14}  This sketching can help us significantly to identify which particular hand gesture is being performed in the time segment 
\subsubsection{EES Datasets: EDA and PPG Sensor Datasets}
We used Eight-Emotion Sentics (EES) dataset to validate \emph{AutoCogniSys} proposed physiological signal processing approaches \cite{picard01}  The dataset consists of measurements of four physiological signals (PPG/Blood Volume Pulse, electromyogram, respiration and Skin Conductance/EDA) and eight affective states (neutral, anger, hate, grief, love, romantic love, joy, and reverence)  The study was taken once a day in a session lasting around 25 minutes for 20 days of recordings from an individual participant  We consider only PPG and EDA for all of the affective states in our study 
\subsubsection{Baseline Methods}
Though no frameworks ever combined all modalities together into real-time automated cognitive health assessment, we evaluate \emph{AutoCogniSys} performance by comparing the performances of its components individually with upto date relevant works  For hand gesture and postural activity recognition, we consider \cite{alam17} proposed method as baseline  For complex activity recognition, we compare our hand gesture and postural activity classifiers aided HDBN model with three-level Dynamic Bayesian Network \cite{zhu12} framework  For activity performance estimation, activity performance based cognitive health assessment; and EDA and PPG based cognitive health assessment, we have considered \cite{alam16} proposed method as baseline 
\subsection{Activity Recognition Evaluation}
The standard definition for \emph{accuracy} in any classification problem is $\frac{TP+TN}{TP+TN+FP+FN}$ where $TP,TN,FP$ and $FN$ are defined as true positive, true negative, false positive and false negative  For complex activity recognition evaluation, we additionally consider \emph{start/end duration error} as performance metric that can be explained as follows: consider that the true duration of ``cooking'' is 30 minutes (10:05 AM - 10:35 AM) and our algorithm predicts 29 minutes (10 10 - to 10 39 AM)  Then, the start/end duration error is 9 minutes ($|$5 minutes delayed start$|$ + $|$4 minutes hastened end$|$), in an overall error of e g , 30\% (9/30=0 3)  We measure cross-participant accuracy using leave-two-participants-out method for performance metrics, i e , we take out two of the participants' data points from the entire dataset, train our proposed classification models, test the model accuracy on the two left-out participants relevant data points, and continue the process for entire dataset 

\begin{figure*}[!htb]
\begin{minipage}{0 45\textwidth}
\begin{center}
   \epsfig{file=hand_gesture_accuracy pdf,height=1 6in, width=3in}
\caption{Feature Weighted Naive Bayes (FWNB) classification accuracy comparisons with baseline approaches (graphical signatures of all hand gestures are shown) }
   \label{fig:hand_gesture_accuracy}
\end{center}
\end{minipage}
\begin{minipage}{0 29\textwidth}
\begin{center}
\vspace{- 12in}
   \epsfig{file=posture_accuracy_normal pdf,height=1 6in, width=2 1in}
\caption{4-class postural level activity recognition performance and comparisons with baseline method}
   \label{fig:posture_accuracy_normal}
\end{center}
\end{minipage}
\begin{minipage}{0 25\textwidth}
 \begin{center}
 \vspace{- 12in}
   \epsfig{file=posture_accuracy_extended pdf,height=1 6in, width=2 1in}
\caption{6-class diverse postural activity recognition framework accuracy comparisons with the baseline approach }
   \label{fig:posture_accuracy_extended}
\end{center}
 \end{minipage}
\end{figure*}

Fig~\ref{fig:hand_gesture_accuracy} displays Feature Weighted Naive Bayes (FWNB) based the 8-hand gestural activity recognition accuracies comparisons with the baseline methods which clearly depicts the outperformance of our method (5\% improvement) with an overall accuracy of 92\% (FP rate 6 7\%) in RCC dataset  For postural activity recognition, dataset achieving 91\% postural activity recognition accuracy (FP rate 9 5\%) which outperforms the baseline approach significantly (8\% improvement)  Now, we expand the postural activities for RCC datasets into 3 diverse `walking' postures: `normal walking', `walking with walker', `walking with single stick' and the accuracy goes down to 88\% (FP 7 9\%)  Fig ~\ref{fig:posture_accuracy_normal} and Fig ~\ref{fig:posture_accuracy_extended} illustrate 4-class postural and extended 6-class postural classifier accuracies respectively which clearly posit that \emph{AutoCogniSys} outperforms in each case of postural activities as well as overall performances (8\% and 7\% improvement respectively) 

For complex activity classification, we choose RCC dataset to train our HDBN model  Our leave-two-participants out method results an accuracy of 85\% (FP Rate 3 6\%, precision 84 2\%, recall 84 5\%, ROC Area 98 2\%) with a start/end duration error of 9 7\%  We run the entire evaluation for baseline complex activity recognition algorithm too achieving an overall accuracy of 78\% (FP Rate 5 2\%, precision 79 6\%, recall 78 5\%, ROC Area 82 7\%) which is clearly lower performed method than our approach  Fig  \ref{fig:complex_activity_roc} and Fig~\ref{fig:complex_activity_accuracy} illustrate the ROC curve and each complex activity recognition accuracy comparisons with baseline method which depict the outperformance of our framework over baseline methods (7\% improvement)  Fig~\ref{fig:complex_activity_accuracy} also shows that inclusion of postural activity improves the final complex activity recognition (4\% improvement) 
 \begin{figure}  [!htb]
  \begin{minipage}{0 15\textwidth}
 \begin{center}
   \epsfig{file=complex_activity_roc pdf,height=1 4in, width=1 1in}
\caption{ROC curve for complex activity recognition}
   \label{fig:complex_activity_roc}
\end{center}
\end{minipage}
\begin{minipage}{0 33\textwidth}
\begin{center}

   \epsfig{file=complex_activity_accuracy pdf,height=1 4in, width=2 3in}
\caption{Complex ADLs recognition accuracy improvement and comparison with baseline \cite{zhu12} and HMM based method}
   \label{fig:complex_activity_accuracy}
\end{center}

\end{minipage}
\end{figure}

\subsection{Quantification of Performance Score}
To characterize both the qualitative and quantitative health assessment performance scores, we start with four different feature groups ranging from both functional and physiological health measures: (i) observation based activity features, (ii) automatic activity performance features, (iii) EDA features and (iv) PPG features 

In \emph{observation based activity features}, we design a complex activity set comprised of multiple subtasks which are involved with task {\it interruption, completion and sequencing}  Participants are instructed to perform the complex activities while the trained evaluator observed the aforementioned functional activity performance measures  Each incorrect attempt of performance measure will be assigned one point thus higher score reflects lower performance of functional activities \cite{dawadi14}  We first detect hand gesture and postural activities  Then, we feed the low-level activity contexts (gestural and postural) combined with ambient contexts (object and ambient motion sensor readings) into HDBN for single inhabitant model \cite{alam16b} to recognize complex activities  The complex activity recognition framework provides both activity labels and activity window (start-end points)  Then, we extract features of object sensor, ambient sensor, gestural activity and postural activity events for each activity window  The features are number of occurrences, mean number of occurrences, consecutive 1, 2, 3, $\ldots$ 20 occurrences, top 10, 20, 30, $\ldots$, 90 percentile etc (29 features in total)  In \emph{physiological features} we first detect 13 complex activities using HDBN algorithm which provides activity labels and activity window (start-end points), apply noise reduction, motion artifacts removal, extract 7 EDA features and 8 HRV features for each activity and take the mean of them over time (minutes) to get 15 (7+8) complex activity physiological features set for each participant  In summary, we extract 3 observation based activity features, 29 automatic activity performance features, 7 EDA features and 8 HRV features 
\subsection{Physiological Signal Processing Performance Evaluation}
Standard evaluation technique should use both experimental and publicly available datasets to confirm the outperformance of the novel approaches  We first evaluate our physiological signal processing techniques using a publicly available dataset (EES Dataset \cite{picard01}) to detect 8 human emotions  Then, in next section, we evaluate our methods in assessing cognitive health status of older adults using RCC dataset 

For EDA, we first apply SWT method to remove motion artifacts and noises  Then, we use cvxEDA method to separate tonic and phasic components of EDA  Then, we extract 7 EDA features on a sliding window of 4 seconds  Finally, we feed the 7 EDA features into a SMO based SVM algorithm \cite{cao06}  We use 10-fold cross validation to classify eight emotions achieving 87\% of overall accuracy (FP rate 6\%)  For PPG, we first apply our proposed PMAF based noises and motion artifacts removal technique  Then, we calculate HRV and perform time-domain feature extraction to extract 8 HRV features on a sliding window of 4 seconds  We feed these features into a SMO based SVM algorithm \cite{cao06}  Our 10-fold cross validation shows accuracy of 79\% (FP rate 11 5\%) of detecting 8 emotions on EES Dataset  Fig  \ref{fig:ees_eda} and Fig  \ref{fig:ees_ppg} clearly depict that \emph{AutoCogniSys} proposed EDA and PPG signal processing techniques significantly improve the accuracy over the baseline \cite{alam16} method (10\% and 12\% improvement) 

\begin{figure}[!htb]
\begin{minipage}{0 24\textwidth}
\begin{center}
   \epsfig{file=ees_eda pdf,height=1 2in, width=1 8in}
\caption{(EES Databaset) EDA features based Eight Emotion classification accuracy comparisons with baseline method}
   \label{fig:ees_eda}
\end{center}
\end{minipage}
\begin{minipage}{0 23\textwidth}
\begin{center}
   \epsfig{file=ees_ppg pdf,height=1 2in, width=1 7in}
\caption{(EES Dataset) PPG features based 8-Emotion classification accuracy comparisons with baseline method}
   \label{fig:ees_ppg}
\end{center}
\end{minipage}

\end{figure}
\subsection{Evaluation of Performance Scores}
The feature subsets used in the experimentation for observation and survey based clinical assessments and technology guided physiological and activity initiated health assessments are depicted in Table~\ref{tab:feature_subset}  From our 6 demographics surveys, we find significant distributions in terms of cognition only for SLUMS Score (S-Score)  Based on that, we divide our participants pool into three groups: \emph{Not Cognitively Impaired (NCI), Mild Cognitively Impaired (MCI) and Cognitively Impaired (CI)} where the number of participants are $5$, $7$ and $10$ respectively 
\begin{table}[!t]
\begin{scriptsize}


{\centering 
\renewcommand{\arraystretch}{ 6}
\caption{Feature Subsets}
\label{tab:feature_subset}
\begin{tabular}{|l|L{5 5cm}|}
\hline
\bfseries Feature& \bfseries Description\\
\hline
Observation & Task Completeness (TC), Sequencing (SEQ), Interruptions (INT)\\
\hline
Survey & SLUMS Score (S-Score), ZUNG Score (Z-Score), IADL Score (I-Score), Yale Score (YPAS), Barthel Score (B-Score), GDS Score (G-Score)\\
\hline
EDA  and HRV & 7 and 8 Features\\
\hline
Activity Performance& Supervised (TC, SEQ, INT), Unsupervised\\
\hline
Arousal& EDA and HRV features of each complex activity window\\
\hline

\end{tabular}
}
\end{scriptsize}
\end{table}


\begin{figure}[!htb]
\begin{center}
   \epsfig{file=group_correlation pdf,height=1in, width=3 3in}
\caption{\emph{AutoCogniSys} Proposed Method Based Group Correlation analysis ( $r-value$) NCI, MCI and CI represent not cognitive, mild cognitive and cognitively impaired group of population  TC, INT, SEQ, EDA and HRV represent task completeness, interruption scores, sequencing scores, electrodermal activity features and heart rate variability features}
   \label{fig:group_correlation}
\end{center}
\vspace{- 2in}
\end{figure}
\begin{figure}[!htb]
\begin{center}
   \epsfig{file=group_correlation_baseline pdf,height=1in, width=3 3in}
\caption{Baseline \cite{alam16} method based Group Correlation analysis ( $r-value$)}
   \label{fig:group_correlation_baseline}
   \vspace{- 25in}
\end{center}
\end{figure}

\subsection{Statistical Correlation Analysis of Cognitive Health}
We used Pearson correlation coefficients with significance on $p<0 05$* for individual feature and partial correlation coefficients with significance on $p<0 005$** for group of features correlation analysis  Fig  \ref{fig:group_correlation} and Fig  \ref{fig:group_correlation_baseline} show the group correlation analysis results based on \emph{AutoCogniSys} proposed framework and baseline \cite{alam16} framework respectively  It can be clearly depicted that our proposed framework improves the correlation with the ground truths 
\subsection{Machine Learning Classification of Cognitive Health}
We evaluate using machine learning classifiers to predict cognitive status of older adults using both individual modalities and combined features  We use leave-two-participants out method to train and test classification accuracy 

We first choose the individual activity features (machine learning method based interruption scores, sequencing scores, unsupervised scores) and their combined features to train and test cognitive impairment status classification for SMO based SVM algorithm \cite{cao06}  The classification accuracies are 72\%, 69\%, 76\% and 83\% respectively  Then we consider 7 EDA-activity features and 8 HRV-activity features individually in training and testing phase of SMO based SVM algorithm \cite{cao06} resulting 85\% and 80\% accuracy respectively 

\begin{figure}[!htb]
\begin{minipage}{0 24\textwidth}
\begin{center}
   \epsfig{file=combined_classification pdf,height=1 2in, width=1 7in}
   \vspace{- 15in}
\caption{Individual and combined classification accuracies comparison with baseline method for cognitive impairment status detection}
   \label{fig:combined_classification}
\end{center}
\end{minipage}
\begin{minipage}{0 23\textwidth}
\begin{center}
   \epsfig{file=each_activity_cognitive_assessment pdf,height=1 2in, width=1 7in}

\caption{Machine learning based cognitive health assessment accuracy for each complex activity in terms of activity, EDA and HRV features }
   \label{fig:each_activity_cognitive_assessment}
\end{center}
\end{minipage}
\end{figure}

For combined classifier, we first applied sequential forward feature selection to find the best combinations of 1- 3 features for cognitive impairment classification group MCI, NCI and CI in terms of combined activity features (29 features), EDA-activity features (7 features) and HRV-activity features (8) features  Our final combined classifier (SMO based SVM algorithm \cite{cao06}) provides an accuracy of {\bf 93\%} in detecting the cognitive impairment status of older adults  Fig  \ref{fig:combined_classification} shows our proposed individual and combined methods outperform the baseline \cite{alam16} significantly (13\% improvement)  Fig  \ref{fig:each_activity_cognitive_assessment} shows the cognitive impairment status prediction accuracy for each modality (activity feature, EDA and HRV) per individual complex activity 
\subsection{Discussion}
If we exclude the postural activities from automated activity performance scoring, we find reduced statistical correlation with original task completeness performance for \{NCI, MCI, CI\} participant group (INT 0 53*, SEQ 0 21' and unsupervised 0 49')  However, if we skip our proposed motion artifact removal stage, we find reduced statistical correlation with \{NCI, MCI\} and \{MCI, CI\} groups of participants (EDA and HRV correlations respectively \{0 51*, -0 51*\} and \{-0 53*,0 46\})  To test our proposed motion artifacts removal impact on EDA signals more rigorously, we choose 5 random participants, engage one expert motion artifact annotator to annotate motion artifacts segment on each participant's first 30 minutes of complex dataset using recorded video and apply both baseline and our methods to detect motion artifact segments  While baseline method achieves 75 5\% (FP rate 20 3\%) accuracy in detecting motion artifact segments, \emph{AutoCogniSys} outperforms achieving 89 9\% (FP rate 8 9\%) accuracy  In terms of experience, we have seen 100\% acceptance of wearing wrist-band,  71\% of acceptance for signing consent on using cameras and 0\% failure rate of collecting continuous data 
\section{Conclusion}
We propose, \emph{AutoCogniSys}, an IoT inspired design approach combining wearable and ambient sensors embedded smart home design, extensive signal processing, machine learning algorithms and statistical analytics to automate cognitive health assessment in terms of complex activity performances and physiological responses of daily events  Additionally, our postural activity detection approach in diverse population cum improved activity performance measurement and fundamental physiological sensor artifacts removal from physiological sensors help facilitate the automated cross-sectional cognitive health assessment of the older adults  Our efficient evaluation on each modality (physical, physiological and ambient) and each activity mode proves that any of the mode (say single activity and single sensor) also can provide significant improved cognitive health assessment measure 




Context 3: 2015-05-14 Assigned to ROVI GUIDES, INC  reassignment ROVI GUIDES, INC  MERGER (SEE DOCUMENT FOR DETAILS)  Assignors: TV GUIDE, INC 
2015-05-14 Assigned to UV CORP  reassignment UV CORP  MERGER (SEE DOCUMENT FOR DETAILS)  Assignors: UNITED VIDEO PROPERTIES, INC 
2015-05-14 Assigned to TV GUIDE, INC  reassignment TV GUIDE, INC  MERGER (SEE DOCUMENT FOR DETAILS)  Assignors: UV CORP 
Methods and systems are described herein for quickly and easily displaying supplemental information about an event occurring in a media asset  In some embodiments, a media application may use a content-recognition module to determine the context of an event and distribute itemized tasks to multiple entities in order to generate the supplemental information about the event 
While viewing media assets (e g , a television program), users may wish to learn more information about an event (e g , a statement made by a person appearing in the media asset, the validity of a claim in an advertisement, etc ) occurring in the media asset  While some media assets allow a user to select additional options or added features (e g , pop-up biographies about the cast and crew), when the added features appear and what topic the added features concern are determined by the content producer and not the user  Furthermore, as the added feature is derived from the content producer, the added feature may be biased or may present limited viewpoints about an event  Therefore, added features provided by a content producer may not provide the added information about an event that a user desires 
In order to gain the added information that a user desires, the user may use additional devices (e g , a laptop computer) to search (e g , using an Internet search engine) for more information about the event  However, without knowing the proper context (e g , who said the statement, what was the tone of the statement, when was the statement said, etc ) of the event or what search terms to use to describe the context of the event (e g , how to describe the tone of the statement), a user may not be able to determine (even using a search engine) more information about the event  Moreover, the use of general search terms may not provide the accuracy or precision needed by the user  Furthermore, even if a user may eventually determine the information, the effort and time required may distract the user from the media asset 
Accordingly, methods and systems are described herein for quickly and easily displaying supplemental information about an event occurring in a media asset  In some embodiments, a media application may use a content-recognition module to determine the context of an event in a media asset and distribute itemized tasks to multiple users in order to generate the supplemental information about the event  The context-recognition module prevents the user from being distracted from the media asset (e g , while the user attempts to describe the context of the event or search for information about the event)  In addition, by distributing tasks to multiple entities (e g , crowd-sourcing), the media application may collect large amounts of information in relatively short periods of time (or in real-time) and aggregate and/or filter the information to generate the supplemental information about the event based on multiple viewpoints and/or sources  By using multiple viewpoints and/or sources, the media application enhances the completeness (e g , by providing unbiased information) and accuracy of the supplemental information 
For example, when a statement or action is made by a character or person appearing on a media asset (e g , a television program), a user may request supplemental information about the statement or action  In response, the media application may determine the context of the statement (e g , who said the statement and to what the statement was referring) or action (e g , what was the reason for the action)  After determining the context of the statement or action, the media application may itemize into tasks the additional information it requires in order to generate the supplemental information  The media application may then transmit requests including the tasks to a plurality of other users  Based on the responses from the plurality of other users, the media application may generate the supplemental information for display to the user 
In some embodiments, a media application may use multiple types of content-recognition modules and/or algorithms to determine the context of an event  For example, the media application may process data associated with the event in order to determine the context of an event  In some embodiments, processing the various types of data may include cross-referencing the data in a database indicating the different contexts the event may have 
In some embodiments, a media application may generate supplemental information about an event in a media asset in response to a user request  In order to generate the supplemental information, the media application may transmit, to multiple users, a request for additional information regarding a context of an event shown in a media asset  Upon receiving messages from the plurality of users that include the requested additional information, the media application may generate the supplemental information associated with the context of the event based on the messages 
It should be noted, the systems and/or methods described above may be applied to, or used in accordance with, other systems, methods and/or apparatuses 
FIG  9 is a flowchart of illustrative steps for generating supplemental information based on additional information provided by a plurality of users in accordance with some embodiments of the disclosure 
Accordingly, methods and systems are described herein for quickly and easily displaying supplemental information about an event occurring in a media asset  The methods and systems described herein alleviate the need for a user to determine the proper context (e g , who said a statement, what was the tone of the statement, when was the statement said, etc ) of an event in a media asset, or the search terms to use to describe the event (e g , the proper search terms to describe the tone of the statement), in order to determine more information about the event  In addition, the methods and systems increase the completeness and accuracy of the information compared to information gathered using traditional searching methods (e g , an Internet search engine), without distracting the user from the media asset 
In some embodiments, a media application may receive a user input from a user device for supplemental information about the context of an event shown in a media asset  The media application may determine additional information required to generate the supplemental information about the context of the event shown in a media asset, and transmit requests for the additional information to one or more users  The media application may receive one or more messages, which include the requested additional information, from the one or more users and generate the supplemental information based on the one or more message  The media application may then instruct the user device to display the supplemental information 
As used herein, “supplemental information” refers to any information related to or associated with an event in a media asset  For example, supplemental information may include, but is not limited to, the verification of a statement or claim in a media asset, further descriptions and/or information about objects or entities shown and/or described in a media asset, and/or any other information, including, but not limited to, a video or audio segment, that may interest a user about an event in a media asset  In some embodiments, the media application may generate supplemental information based on one or more pieces of additional information 
As used herein, “additional information” refers to any information used to generate supplemental information  For example, in an embodiment in which supplement information is the verification of a statement made by a person displayed in a media asset, and a request for the additional information from the media application includes a request for a fact needed to verify the factual basis of the statement, the additional information may be the fact used to verify the statement  For example, if an advertisement claims to have the best product on the market, the media application may use additional information such as the name of the product in question, a list of all other products in the market, and the results of a comparison study of the product in question to all other products to determine whether or not the product is actually the “best” product on the market  Additionally or alternatively, the media application may request industry and/or user reviews related to the event (e g , reviews indicating the quality of the product)  The media application may then use the information in the reviews to generate the supplemental information 
As used herein, an “event” is any action (e g , a verbal statement, opinion and/or physical movement), segment (e g , a portion of a news broadcast featuring a particular topic), or other occurrence during a media asset that may be of particular interest to a user  For example, in some embodiments an event may be a statement or gesture made by a character or person in a media asset affirming or denying a claim 
As referred to herein, the terms “media asset” and “content” should be understood to mean an electronically consumable user asset, such as television programming, as well as pay-per-view programs, on-demand programs (as in video-on-demand (VOD) systems), Internet content (e g , streaming content, downloadable content, Webcasts, etc ), video clips, audio, content information, pictures, rotating images, documents, playlists, websites, articles, books, electronic books, blogs, advertisements, chat sessions, social media, applications, games, and/or any other media or multimedia and/or combination of the same  Media applications also allow users to navigate among and locate content  As referred to herein, the term “multimedia” should be understood to mean content that utilizes at least two different content forms described above, for example, text, audio, images, video, or interactivity content forms  Content may be recorded, played, displayed or accessed by user equipment devices, but can also be part of a live performance 
As referred to herein, the phrase “user equipment device,” “user equipment,” “user device,” “electronic device,” “electronic equipment,” “media equipment device,” or “media device” should be understood to mean any device for accessing the content described above, such as a television, a Smart TV, a set-top box, an integrated receiver decoder (IRD) for handling satellite television, a digital storage device, a digital media receiver (DMR), a digital media adapter (DMA), a streaming media device, a DVD player, a DVD recorder, a connected DVD, a local media server, a BLU-RAY player, a BLU-RAY recorder, a personal computer (PC), a laptop computer, a tablet computer, a WebTV box, a personal computer television (PC/TV), a PC media server, a PC media center, a hand-held computer, a stationary telephone, a personal digital assistant (PDA), a mobile telephone, a portable video player, a portable music player, a portable gaming machine, a smart phone, or any other television equipment, computing equipment, or wireless device, and/or combination of the same 
In some embodiments, the user equipment device may have a front facing screen and a rear facing screen, multiple front screens, or multiple angled screens  In some embodiments, the user equipment device may have a front facing camera and/or a rear facing camera  On these user equipment devices, users may be able to navigate among and locate the same content available through a television  Consequently, media may be available on these devices, as well  The media provided may be for content available only through a television, for content available only through one or more of other types of user equipment devices, or for content available both through a television and one or more of the other types of user equipment devices  The media applications may be provided as on-line applications (i e , provided on a website), or as stand-alone applications or clients on user equipment devices  Various devices and platforms that may implement media applications are described in more detail below 
In some embodiments, a media application may transmit, to a plurality of users, a request for additional information regarding a context of an event shown in a media asset  As used herein, a “plurality of users” may include, but is not limited to any device, entity, or source of information that may process a request for additional information  For example, the plurality of users may include a person operating a user equipment device  In some embodiments, the person may receive (e g , via e-mail, Internet posting, advertisement, or any other applicable information delivery method) the request from the media application for additional information, and in response generate a message (e g , via a return e-mail, an answer to the Internet posting, a user input in the advertisement, or any other applicable method of transmitting information) that includes the additional information  It should be noted that in some embodiments, transmitting a request to a plurality of users may also include querying one or more databases (e g , an Internet search engine or any other storage device, including, but not limited to, databases containing previously generated supplemental information and/or additional information) or consulting one or more data gathering services (e g , a intelligent personal assistant application) for the additional information 
In some embodiments, a media application may use a content-recognition module or algorithm to determine the context of an event and distribute itemized tasks to multiple users in order to generate the supplemental information about the event  The content-recognition module may use object recognition techniques such as edge detection, pattern recognition, including, but not limited to, self-learning systems (e g , neural networks), optical character recognition, on-line character recognition (including but not limited to, dynamic character recognition, real-time character recognition, intelligent character recognition), and/or any other suitable technique or method to determine the objects and/or characteristics in media assets  For example, the media application may receive media assets in the form of a video  The video may include a series of frames  For each frame of the video, the media application may use a content-recognition module or algorithm to determine the context (e g , the person that is speaking or a facial gesture affirming or denying a statement) of an event occurring during the frame or series of frames 
In some embodiments, the content-recognition module or algorithm may also include speech recognition techniques, including but not limited to Hidden Markov Models, dynamic time warping, and/or neural networks (as described above) to translate spoken words into text  The content-recognition module may also use other techniques for processing audio and/or visual data  For example, the media application may monitor the volume of a statement in a media asset to determine the tone of the statement (e g , a high volume may indicate an angry tone) 
In addition, the media application may use multiple types of optical character recognition and/or fuzzy logic, for example, when determining the context of a keyword(s) retrieved from data (e g , media data, translated audio data, subtitle data, user-generated data, etc ) associated with the media asset (or when cross-referencing various types of data with databases indicating the different contexts of events as described below)  For example, the particular data field may be a textual data field  Using fuzzy logic, the system may determine two fields and/or values to be identical even though the substance of the data field or value (e g , two different spellings) is not identical  In some embodiments, the system may analyze particular data fields of a data structure or media asset frame for particular values or text  The data fields could be associated with characteristics, additional information, and/or any other data required for the function of the embodiments described herein  Furthermore, the data fields could contain values (e g , the data fields could be expressed in binary or any other suitable code or programming language) 
As used herein, the “context” of an event refers to the set of circumstances or facts that surround a particular event that influence or affect the meaning of the event  For example, when determining the context of a written and/or spoken statement, the media application may determine who or what authored/stated the statement, the written and/or spoken words and/or other statements that preceded and/or followed the statement, the tone of the statement, and/or any other conditions that may alter the connotation of the statement 
FIG  1 shows an illustrative example of a media application that may be used to display supplemental information in accordance with some embodiments of the disclosure  Display 100 illustrates a display on a user device displaying a media asset  Display 108 illustrates a display featuring supplemental information as described and/or generated in FIGS  6-9  It should be noted that display 100 and display 108 may be presented on any of the devices shown in FIGS  3-4  For example, in some embodiments, display 100 and display 108 may be displayed on user equipment 402, 404, and/or 406 (FIG  4) 
In FIG  1, display 100 represents a display of a media asset (e g , a streaming television program) on a user device (e g , user equipment 402, 404, and/or 406 (FIG  4))  Display 100 includes entity 102 and entity 104  In display 100, entity 104 is currently speaking as indicated by event 106  As shown in FIG  1, event 106 is a statement (e g , “We export a lot of coal”) by a person in the media asset 
In some embodiments, display 108 represents the continued display of the media asset on a user device, after a user has requested supplemental information about event 106  For example, a media application may have received a user input (e g , via user input interface 310 (FIG  3)) while entity 104 was speaking  Using the systems and methods described herein (e g , FIGS  6-9), the media application generated supplemental information 110  Supplemental information 110 represents more information about event 106 
For example, the media application (e g , media application 206 (FIG  2)) may have determined the context of event 106  Specifically, the media application may determine via a content-recognition module or algorithm the words spoken and/or actions by the person during the event  Additionally or alternatively, the media application may analyze the words and/or action during a predetermined amount of time (e g , ten seconds) before and/or after the event (e g , in order to better understand the context of the event)  Furthermore, by cross-referencing the words and/or other information obtained by the content-recognition module (e g , as discussed below in relation to FIG  5) with a database, the content-recognition module determines that the term “we,” the person in the media asset refers to an organization or body  The content-recognition module or algorithm may also determine that the term “export” refers to shipping goods out of a country  The content-recognition module or algorithm may also determine that the term “a lot” refers to a particular numerical amount  Finally, the content-recognition module or algorithm may also determine that the term “coal” refers to a mineral of fossilized carbon 
The content-recognition module or algorithm may also determine the relationships between words and/or other information obtained by the content-recognition module  For example, by processing the relationship between the words, the media application determines that event 106 is a statement regarding a particular amount of a particular substance shipped out of a particular country  Therefore, the media application determines that the request for supplemental information is likely a request to determine the validity of the statement  The media application then generates the supplemental information 
The media application may also have stored supplemental information generated by previous requests (e g , supplemental information generated in response to the same or different user viewing the media asset at an earlier date), and display the supplemental information again during the event (either in response to a user input requesting supplemental information or automatically without a user requesting supplemental information) 
FIG  2 shows an illustrative example of a system that may be used to generate supplemental information (e g , supplemental information 110 (FIG  1)) based on additional information provided by a plurality of users in accordance with some embodiments of the disclosure  For example, in some embodiments, system 200 may be used to generate supplemental information (e g , supplemental information 110 (FIG  1)) on a display (e g , display 108 (FIG  1)) of a user device (e g , user equipment 402, 404, and/or 406 (FIG  4))  It should be noted that in some embodiments, the devices shown in FIG  2 may correspond to one or more devices in FIGS  3-4 
FIG  2 shows system 200  In system 200, a user is currently accessing a media asset on display 202  In some embodiments, display 202 may correspond to display 100 (FIG  1))  During an event (e g , event 106 (FIG  1)) a user may have requested supplemental information about an event (e g , event 106 (FIG  1)) in display 202 using user device 204  Media application 206, which in some embodiments, may be implemented on user device 204 or at a remote location (e g , supplemental information source 418 (FIG  4)), receives the request for supplemental information 
Media application 206 determines the context of the event (e g , who said the statement making up the event and to what the statement was referring)  After determining the context of the statement, the media application may itemize into one or more tasks, additional information (e g , facts) it requires in order to generate the supplemental information (e g , a verification or correction of the factual basis of the statement)  For example, if the event is a statement about the amount of coal that is exported from the United States (e g , as described in relation to FIG  1 above), media application 206 may determine the fact required to generate the supplemental information is the exact numerical amount of coal that is exported from the United States  The media application may then transmit requests for the additional information (e g , a request for the exact numerical amount of coal that is exported from the United States) to a plurality of other users 
In FIG  2, users operating user device 208, user device 210, and user device 212 represent a plurality of users  Having determined the additional information it requires in order to generate the supplemental information, media application 206 requests the additional information from the plurality of users  In system 200, media application 206 has transmitted the same task (e g , the same question) to each of the plurality of users  In some embodiments, one or more of the users may receive different tasks  For example, by breaking the additional information into small, independent tasks, media application 206 may increase the speed (e g , multiple users may work concurrently to solve different parts of a problem) and accuracy (e g , reducing the tasks to smaller, less complex problems reduces the chance of human error) of the additional information returned by the plurality of users 
In addition, by breaking the additional information into small, independent tasks, the plurality of users may not know to what they are contributing (enhancing the privacy of the user that requested the supplemental information), however, the plurality of users can still be effective in their individual tasks  In addition, by breaking the additional information into small, independent tasks, the media application may more easily outsource the requests for additional information  For example, one or more of the tasks used to generate the additional information may be the same as one or more of the tasks used to generate other additional information (e g , additional information used to generate different supplemental information in response to a request for supplemental information about the same or a different event issued by the same or a different user)  The response to each of the request and/or the additional information may be stored (e g , on any of the devices accessible by communications network 414 (FIG  4)) for subsequent retrieval 
Based on the responses, transmitted as messages including the additional information, from the plurality of other users, media application 206 may generate the supplemental information (e g , supplemental information 110 (FIG  1)) for display to the user on the user device 204  For example, media application may aggregate, append, and/or compare the additional information in each of the messages received from the plurality of users  The supplemental information may then be generated based on the aggregated, appended, and/or compared additional information (e g , as described in FIG  9 below) 
In some embodiments, the plurality of users may receive summary information about the event with the request for additional information  (e g , a video clip of a portion or segment of the media asset, a textual description, etc ), which may help the plurality of users provide additional information  For example, in some embodiments, the media application may instead of (or in addition to) determining the context of an event, determine a particular portion of the event that would be needed for the plurality of users to provide additional information about the event 
For example, the media application may use progress information associated with the progress of the media asset (e g , line 506 (FIG  5)) to determine at what point during the progression of the media asset the event occurred, and in response, transmit a portion of the media asset beginning ten second before that point and ending ten seconds after that point  For example, if the event is a statement made by a character or person in a media asset, the media application may determine when the statement began (e g , the point of progress of the media asset in which the statement began) and ended  The media application may then include the portion containing the entire statement (and the event) in the request for additional information sent to the plurality of users 
The selected portion may include any amount of summary information that the media application determines is necessary for the user or any one of the plurality of users to understand the main action sequence  This summary information (e g , a portion of the media asset) may be included with the request for additional information (e g , in a file transmitted with the request), or may be included with the generated supplemental information as a reference for the user  For example, the media application may select a segment of the play length of the media asset or a particular scene of the media asset, which includes the event, for to display to the plurality of users along with the request for additional information 
For example, if an event (e g , a statement) was in response to a question, the media application may also determine when the question began and ended, and send the entire question (or the play length of the media asset corresponding to the question) to the plurality of users as well  After determining the portion to provide to the plurality of users (e g , a segment including the ten seconds before and the ten seconds after the event), the media application may provide the summary information of the event and any other material needed by the plurality of users to understand the event and/or request for supplemental information from the user 
In some embodiments, a portion of the media asset containing the event, as selected by the media application, may also include any amount of the play length of the media asset, or any amount of scenes or segments from the media asset  In some embodiments, the portion may include segments of the play length of the media asset or scenes from the media asset that are not adjacent during the normal playback of the media asset  For example, in some embodiments, a portion of the media asset may include one or more sequences or scenes of interest to the plurality of users, even though the particular sequences or scenes are featured at different points in the play length of the media asset  The media application may determine the segments or scenes to include based on a content recognition file (e g , data structure 500 (FIG  5)) describing the media asset  For example, if a plot point or other information, which may be relevant to an event is displayed earlier in the media asset, the summary information may include a portion of the media asset displaying the plot point 
In some embodiments, the length of a portion may be determined based on the genre of the media asset  In some embodiments, the length of the portion may depend on a user profile for the user or for anyone of the plurality of users  For example, a user profile and/or a content recognition file (e g , data structure 500 (FIG  5)) may indicate that a particular user may require more or less additional content  For example, the user may be aware of particular characters or plot points in the media asset and, therefore, may not require the additional content to introduce those aspects 
In some embodiments, the plurality of users may receive a particular user interface, which organizes the data about the event (e g , a clip of the actual event, summary information about the event, information about the request for supplemental information issued by the user, etc )  The interface may also include an automatic submission form, which may be used to generate a message, which is sent to the media application 
In some embodiments, the media application may also receive user input from the user requesting the supplemental information that further affects the generation of supplemental information by the media application  For example, the user may request the supplemental information includes particular information (e g , the factual basis of a statement), may request a multimedia format of the supplemental information (e g , textual description, a video clip, etc ), may request a form of the supplemental information (e g , a short description about the event, an Internet link to other sources of information on the event, or a true or false designation about the event) by entering user inputs (e g , via user input interface 310 (FIG  3)) 
It should be noted that any information or process referred to in this disclosure that is referred to as being in response to a user input may alternatively and/or additionally be performed automatically by the media application (e g , via control circuitry 304 (FIG  3))  For example, in some embodiments, a user may request a true or false designation (e g , an on-screen pop-up box indicating whether an event was true or false)  Additionally and/or alternatively, in some embodiments, the true or false designation may appear automatically based on predetermined settings indicating to the media application to display a true or false designation in response to detecting an event 
In some embodiments, an indicator that supplemental information has previously been generated or is currently ready to generate (e g , a plurality of users are available) may be displayed to a user (e g , on display 100 (FIG  1) during the event)  The indicator may also indicate the particular information, the multimedia format, and/or the form of supplemental information that is available  An indicator may also appear with the supplemental information (e g , supplemental information 110 (FIG  1)), which allows the user to request additional supplemental information or provide feedback/responses (e g , rating the quality of the supplemental information) to the media application and/or plurality of users 
In some embodiments, a user may also access (e g , via selection of an indicator and/or automatically upon the supplemental information being generated) summary information about the event  For example, in some embodiments (e g , when the supplemental information is not generated in real-time), the media asset may have progressed to a different point by the time the supplemental information is ready for display  Therefore, the media application may need to provide a video clip of the event or other summary information, so that the user remembers about what or why the supplemental information was requested 
FIG  3 is a block diagram of an illustrative user equipment device in accordance with some embodiments of the disclosure  It should be noted that the components shown in FIG  3 may be used to store, receive, transmit, and/or display the media assets, additional information, and/or supplemental information as described herein  For example, media application 206 (FIG  2) may be implemented on user equipment device 300, and may issue instructions (e g , displaying supplemental information 110 (FIG  1)) via control circuitry 304 
Users may access media assets and the media application (and its display screens described above and below) from one or more of their user equipment devices  FIG  3 shows a generalized embodiment of illustrative user equipment device 300  More specific implementations of user equipment devices are discussed below in connection with FIG  4  User equipment device 300 may receive content and data via input/output (hereinafter “I/O”) path 302  I/O path 302 may provide content (e g , broadcast programming, on-demand programming, Internet content, content available over a local area network (LAN) or wide area network (WAN), and/or other content) and data to control circuitry 304, which includes processing circuitry 306 and storage 308  Control circuitry 304 may be used to send and receive commands, requests, and other suitable data using I/O path 302  I/O path 302 may connect control circuitry 304 (and specifically processing circuitry 306) to one or more communications paths (described below)  I/O functions may be provided by one or more of these communications paths, but are shown as a single path in FIG  3 to avoid overcomplicating the drawing 
Control circuitry 304 may be based on any suitable processing circuitry such as processing circuitry 306  As referred to herein, processing circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc , and may include a multi-core processor (e g , dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputer  



ground_truth: Centralized control of mobile devices and applications.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What types of sensors are now capable of estimating physical activity levels and physiological outcomes of older adults?

Read the following text and answer briefly.

Context 4: Paper Info

Title: Two-stage Pipeline for Multilingual Dialect Detection
Publish Date: Unkown
Author List: Ankit Vaidya (from Pune Institute of Computer Technology), Aditya Kane (from Pune Institute of Computer Technology)

Figure

Figure 1: Class distribution of dialects
Figure 2: System diagram for dialect classification The LID classifies the input into one of 3 languages The sample is then further classified into dialects by language specific models 
Figure 3: Confusion matrix of 9-way classification Note that rows are normalized according to the number of samples is that class 
Our complete results for Track-1 using the two-stage dialect detection pipeline Model-* denotes the language of the models used for the experiments 
Performance on Track-1 validation dataset of individual models used in the two-stage pipeline "Lg" stands for language of the model used 
Comparative results of two-way classification using the finetuned (F T ) predictions and predictions adapted from three-way classification models 

abstract

Dialect Identification is a crucial task for localizing various Large Language Models  This paper outlines our approach to the VarDial 2023 DSL-TL shared task  Here we have to identify three or two dialects from three languages each which results in a 9-way classification for Track-1 and 6-way classification for Track-2 respectively 
Our proposed approach consists of a two-stage system and outperforms other participants' systems and previous works in this domain  We achieve a score of 58 54% for Track-1 and 85 61% for Track-2  Our codebase is available publicly 1  

Introduction

Language has been the primary mode of communication for humans since the pre-historic ages  Studies have explored the evolution of language and outlined mathematical models that govern the intricacies of natural language   Inevitably, as humans established civilization in various parts of the world, this language was modified by, and for the group of people occupied by that particular geographical region 
This gave rise to multiple national dialects of the same language  The VarDial workshop (colocated with EACL 2023) explores various dialects and variations of the same language  We participated in the Discriminating Between Similar Languages -True Labels (DSL-TL) shared task  In this task, the participants were provided with data from three languages, with each language having three varieties 
This shared task consisted of two tracks -Track-1 featuring nine-way classification and Track-2 featuring six-way classification  The second track included two particular national dialects of each language (eg  American English and British English), and the first track had one general  We ranked 1 st in both of the tracks 
Moreover, we beat the next best submission by a margin of 4 5% in the first task and 5 6% in the second task We were the only team to surpass the organizer baseline scores  We present our winning solution in this paper  We used an end-to-end deep learning pipeline which consisted of a language identification model and three language-specific models, one for each language 
We converged upon the best combination by doing an elaborate analysis of various models available  Furthermore, in this work we also analyze the performance of the pipeline as a whole and also provide an ablation study  Lastly, we provide some future directions in this area of research 

Related Work

The present literature encompasses various aspects of dialect identification  We study this from three perspectives: large language models, language identification and dialect classification problems 

Large Language Models

The success of transformers and BERT based models was inevitable since the initial boom of the transformer  2017) model  In recent years, many other architectures like RoBERTa and ELECTRA have further pushed the state-of-the-art in this domain  Moreover, autoregressive models like GPT and GPT-2 have also shown their prowess 
Multilingual versions of RoBERTA, namely XLM-RoBERTa are also available  Lastly, language specific models like Spanish BERT (la Rosa y Eduardo G  Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury, 2022) and Portuguese BERT are available as well  Our winning solution makes use of these large language models trained on specific languages 

Language Identification Models

Many multilingual language identification models have been developed in order to classify the language of the input sentence beforehand  Even though the initial works used n-gram models and generative mixture models or even conditional random fields and other classical machine learning methods like naive bayes , modern methods have shifted to the use of deep learning for language identification  
Recent works have mainly focused on deep learning based language identification, where handling codemixed data is a big challenge in the domain  For our experiments, we use a version of XLM-RoBERTa finetuned on a language identification dataset 2   This model has near-perfect test accuracy of 99 6% 

Dialect Classification

Dialect classification has been previously solved using statistical methods like Gaussian Mixture Models and Frame Selection Decoding or Support Vector Machines (SVM)   It has been explored relatively sparsely, mostly in the case for local languages   Deep learning approaches have been explored in previous editions of the VarDial workshop shared tasks and otherwise  
Dialect classification was also explored previously as a part of other shared tasks   We want to stress that given the multilingual nature of the dataset, using the present methods directly was not an option  In our work, although we take inspiration from the previous works, we propose a novel system that surpasses the performance of the previous systems by a large margin 

Data

The dataset  We observed that the class PT-BR had the most number of samples (2,724) and the class EN had the least number of samples (349), and thus the imbalance ratio was almost 1:8  We have illustrated the data distribution in Figure   We tried to mitigate this imbalance using over-sampling and weighted sampling methods 
However, the improved data sampling method did not affect the performance 

System Description

This was a problem of multi-class classification having 9 classes for Track-1 and 6 classes for Track-2  The samples were belonging to 3 languages having 3 varieties each, so the classification pipeline was made in 2 stages  The Language Identification (LID) model which is the first stage classifies the sentence into 3 languages: English (EN), Spanish (ES) and Portuguese (PT) 
The LID is a pretrained XLM-RoBERTa that is fine-tuned for the task of language identification  It is able to classify the input sentence into 20 languages  We classify and separate the samples according to their language  The samples corresponding to the specific languages are then fed into the language specific models for dialect identification 
For dialect identification we have used models like BERT and RoBERTa with a linear layer connected to the pooler output of the models  Then fine-tuning is done on the models for dialect identification   using the samples corresponding to the specific languages  For the task of dialect identification we experimented with several pretrained models like XLM-RoBERTa, BERT, ELECTRA, GPT-2 and RoBERTa 
All models were fine-tuned for 20 epochs with a learning rate of 1e-6 and weight decay 1e-6 with a batch size of 8



ground_truth: Wearable sensors.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: How does a media application determine the context of an event?

Read the following text and answer briefly.

Context 4:  The best performing model checkpoint was chosen according to the epoch-wise validation macro-F1 score  5 Experiments and Results

Experiments using Large Language Models

For the task of Dialect Identification we have tried various language specific models like XLM-RoBERTa, BERT, ELECTRA, RoBERTa and GPT- 2  The base variant of all these models were used and all the models were used through the Hugging-Face library  The pooler output of these models was passed through a linear layer and the models were fine-tuned 
First, we experimented with different models for Track-1  All the models were trained for 20 epochs with learning rate 1e-6, weight decay 1e-6 and a batch size of 8  We used XLM-RoBERTa as the baseline for all 3 languages  The best performing models for the English language were RoBERTa and BERT whereas GPT-2 was the worst performing 
Similarly the language specific versions of RoBERTa and BERT performed well for the Spanish and Portuguese respectively  Overall the worst performing model was GPT-2 across all 3 languages  The validation F1 scores are present in Table   The two best-performing models for every language were chosen for Track-2 
The same procedure as specified above was used and the F1 scores are present in Table   The train and validation F1 scores for 2-class classification are higher for all models as compared to the F1 score of the same models for 3-class classification  This was mainly due to the poor representation and accuracy of classification of the third class 
We observed symptoms of overfitting in all models after 12-15 epochs and the best validation F1 score was obtained in the range of 4-8 epochs 

LID experiments

The pipeline for dialect identification is divided into two parts as the sentences in the dataset belong to different languages  The stages are described in Section 4  The XLM-RoBERTa we have used for language classification has a test accuracy of 99 6% meaning it correctly classifies all input sentences and hence, can be considered as a perfect classifier 
For the final pipeline we experimented using the two best performing models for each language in Track-1 and Track-2  For both the tracks we experimented with all 8 (2 3 ) possible combinations of models and calculated the validation F1 score for the combined validation dataset which had sentences belonging to all languages 
The validation scores for Track-1 and Track-2 are shown in Table and Table respectively  For both the tracks, the three pipelines with the best validation F1 scores were chosen for submission 

Using 3-way classifier as a 2-way classifier

In Track-1, participants are expected to train a classifier which classifies amongst 9 classes, and in Track-2, participants are expected to train a classifier which classifies amongst 6 classes  These 6 classes are a proper subset of the 9 classes from Track-1  Thus, an intuitive baseline for Track-2 is to use the model finetuned for Track-1, whilst considering only the relevant classes for the latter task 
The classes EN , ES and P T , i e  the classes without any national dialect associated with them are not included in Track-2 as compared to Track-1  Thus, we calculate the predictions for the Track-2 validation dataset using the models for Track-1 and exclude the metrics for Track-1 specific classes to get the metrics for this "adapted" 2-way classification 
We show the results of this experiment in Table and observe that, as expected, the adapted 2-way classification performs worse compared to the explicitly finetuned variant 

Results for Track-1 and Track-2

We now present our experiments and their performance for both tracks  Our experiments for Track-1 are described in Table and our experiments for Track-2 are described in Table   The participants were allowed three submissions for evaluation on the test set, so we submitted predictions using the three systems which performed the best on the validation set 
As mentioned in Section 5 2, we performed 2 3 , i e  a total of 8 experiments using the two best models for each language  We observed that RoBERTa base on English, Spanish BERT base on Spanish and Portuguese BERT base performed the best on the testing set for Track-1  The same combination, with RoBERTa base for English, worked best for Track-2 
All of our submissions were the top submissions for each track, which surpassed the next best competitors by a margin of 4 5% and 5 6% for Track-1 and Track-2 respectively 

Ablation of best submissions

We hereby make some observations of our submissions and other experiments  To assist this, we plot the confusion matrices of our best submissions for Track-1 and Track-2 in Figures respectively  Note that these confusion matrices have their rows (i e  true labels axes) normalized according to the number of samples in the class 
Here are observations from our experiments: 1  BERT-based models outperform other models across all languages: We observe that BERT-based models outperform ELECTRA-based and GPT-2-based models, as shown in Table   We speculate this is because of the inherent architecture of BERT, which combines semantic learning with knowledge retention 
This combination of traits is particularly useful for this task  2  Common labels perform the worst across all languages: We observe that the common labels EN , ES and P T perform the worst, both in the individual as well as the two-stage setup  We hypothesize this is because of the absence of dialect specific words, or words that are specific to the geographical origin of the national dialect (for example, "Yankees" for EN-US and "Oxford" for EN-GB) 
3  English models work better than models of other languages: It can be noted from Figures 4 and 3 that the English models have the best performance across all classes  This can be attributed to two reasons: absence of national dialect specific words and lesser pretraining data in the case of Portuguese 
4  British English is most correctly classified class: We can observe that the Spanish or Portuguese models make equal number of mistakes in the case of either national dialect, in the case of Track-2 (see Figure )  However, in the case of English, the label EN − GB is correctly classified for more than 95% of the cases 
We speculate this is because British English involves slightly distinctive grammar and semantics, which help the model separate it from other classes  5  The proposed 2-step method is scalable for multiple language dialect classification: We can strongly assert that the novel 2-step deep learning method for multilingual dialect classification is a scalable method for the task due to two specific reasons: firstly, the multilingual models (like XLM-RoBERTa) might not have the vocabulary as well as the learning capabilities to learn the minute differences between individual dialects 
Secondly, this system can be quickly expanded for a new language by simply adding a language specific dialect classifier, provided the language identification model supports that particular language 

Conclusion

In this paper we propose a two-stage classification pipeline for dialect identification for multilingual corpora  We conduct thorough ablations on this setup and provide valuable insights  We foresee multiple future directions for this work  The first is to expand this work to many languages and dialects 
Secondly, it is a worthwhile research direction to distill this multi-model setup into a single model with multiple prediction heads  The obvious limitation of this system is the excessive memory consumption due to the usage of language specific models  For low resource languages this system is difficult to train and scale 
We hope that these problems will be addressed by researchers in future works 



ground_truth: It uses a content-recognition module or algorithm.