Patent Description:
The present invention relates in general to computing systems, and more particularly, to various testing embedded systems in an environment where a networked system is monitored using a variety of sensors connected in real-time, also known as the Internet of Things (IoT), and where some of the embedded systems are available in a manufactured form, while others are simulated on demand, denoted as Hardware-in-the-Loop as a Service (HiLaaS).

In today's society, consumers, business persons, educators, and others use various computing systems in a variety of settings, many of which involve interconnection of the said systems. With the advent and further miniaturization of integrated circuits, computing systems can be integrated into a wide variety of personal, business, health, home, education, entertainment, travel and other devices. For example, vehicles of every kind are equipped with multiple computing systems. As the use of such data processing devices continue to proliferate throughout society, such devices are being relied upon. In order to have confidence in such devices and, especially, their interactions, testing the properties of such devices is critical.

<NPL> discloses how cloud computing provides new opportunities to achieve the goals of advanced manufacturing. The paper reviews the historical developments of prognosis theories and techniques and projects their future growth enabled by the emerging cloud infrastructure.

<NPL> discloses an overview of Cloud Manufacturing, Additive Manufacturing and relevant domains, as well as presents the historical development of scientific research in these fields.

<NPL> discusses how telephone calls and other calls (such as data calls) are handled in the network.

According to one aspect, there is provided a method, by a processor, for testing networked systems in an Internet of Things (IoT) environment according to claim <NUM>.

According to another aspect, there is provided a system for testing networked systems in an Internet of Things (IoT) network environment according to claim <NUM>.

According to another aspect, there is provided a computer program product according to claim <NUM>.

Various embodiments for testing networked systems in an Internet of Things (IoT) environment are provided.

In order that the advantages of the invention will be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:.

The advent of computers and networking technologies have made possible the intercommunication of people or devices from one side of the world to the other. For example, cloud computing may be provided as a service over the Internet, such as in the form of "Infrastructure as a Service" (IaaS), "Platform as a Service" (PaaS), and/or "Software as a Service" (SaaS). IaaS may typically provide physical or virtual computing devices and/or accessories on a fee-for-service basis and onto which clients/users may load and/or install, and manage, platforms, applications, and/or data. PaaS may deliver a computing platform and solution stack as a service, such as, for example, a software development platform, application services, such as team collaboration, web service integration, database integration, and/or developer community facilitation. SaaS may deploy software licensing as an application to customers for use as a service on demand. SaaS software vendors may host the application on their own clouds or download such applications from clouds to cloud clients, disabling the applications after use or after an on-demand contract expires.

In an additional aspect, the Internet of Things, also known as the Internet of objects, refers to the networked interconnection of everyday objects or the interconnection of computing devices (IoT devices). It is described as a self-configuring wireless network of sensors whose purpose would be to interconnect all things. The loT devices may be uniquely identifiable and may communicate with one or more other loT devices via computing networks to form device configurations. Multiple protocols, domains, and applications may be implemented in the loT devices. IoT devices may be embedded in a variety of physical devices or products, such as home appliances, manufacturing devices, industrial printers, automobiles, thermostats, smart traffic lights, vehicles, buildings, etc. Further, loT computing systems may include, for example, "demand response management" providing the ability to partially control a power load of residential customers such as, for example, where a power load may be remotely controlled. As an additional example, in the transportation industry, elements of an loT computing system may include one or more types of vehicles.

However, current challenges impact various cloud computing and loT computing systems. For example, in the loT computing system it is difficult to deploy hundreds of thousands of sensors (e.g., "smart wires" and on-wire measurements of temperature) at a single time and it may be challenging to also foresee the benefits of the sensors from deploying only a few sensors in a proposed, pilot project. Thus, the present invention preferably seeks to provide a solution to test embedded systems and application in an Internet of Things (IoT) network environment using Hardware-in-the-Loop as a Service (HiLaaS).

In one embodiment, hardware in the loop may be applied in conjunction with simulation engines (e.g., simple simulation engines), with a fixed number of simulated elements, e.g., to testing an engine of a vehicle independent of the rest of the vehicle, while simulating the traction of <NUM> wheels and inputs from <NUM> driving wheel. In loT applications, the numbers of simulated elements tend to vary and tend to be much larger, e.g., many thousand vehicles, and the precision of a result depends on the number of simulated elements. As such, the present invention preferably provides that HiLaaS testing may apply to such networked systems, where the number of elements in the networked system under test changes over time. Accordingly, the mechanism of the illustrated embodiments may provide simulations of a networked system, where some actual or real elements can be instrumented to provide data in real-time, while a time-varying number of further elements may be simulated. For example, in a power system, the HiLaaS testing extends beyond merely testing a topology that may be constant over a time period (e.g., testing the impact of an outage of a power station), both to testing a topology that varies over time (e.g., optimal transmission switching) and to tests considering individual user interruptions (e.g., testing the individual interruptions of a power load caused by a user increasing or decreasing the power loads). As an additional example, the mechanism of the illustrated embodiments may provide the HiLaaS testing the transportation sector by simulating traffic using both global positioning satellites ("GPS") of actual, real vehicles and also time-varying the number of other vehicles (e.g., time-varying the number of other simulated vehicles in the traffic simulation test).

In one embodiment, time-varying a number of entities for the HiLaaS testing of the networked system is critical for long term analysis. For example, the demand across many applications (e.g., power systems, transportation, networking) exhibits stationarity only when very short time periods (e.g., minutes) are considered. Thus, the present invention preferably provides for a number of simulated entities to change in a non-stationary fashion to simulate an application or system over a selected time period (e.g., greater than or equal to <NUM> hours). The number of real entities may also change such as, for example, due to practical reasons (e.g., breakdowns, scaling up the simulations, etc.) if the simulation is to run for longer periods of time (e.g., years in the testing of road-safety of self-driving vehicles).

In one embodiment, the frequencies of the events to simulate may be defined such as, for example, low frequency of event (e.g., <NUM> Hertz "Hz"), while the number of elements in the networked system may be large in size (e.g., thousands of elements). For example, in a transportation application, such as a speed advisory system for a ring-road or high-way of <NUM> kilometers ("km") with three thousand vehicles on the road, each vehicle may stay on the road for an hour, thus allowing for <NUM> frequency of arrivals and departures of vehicles onto the road. In a city-scale speed advisory system, there may be a hundred thousand vehicles on the road network of <NUM> at peak times, with <NUM> frequency of the arrivals and departures. Further in transportation applications, vehicles may not be able to accelerate and decelerate arbitrarily fast, making the suggested frequency a realistic discretization of time for many other events as well.

The HiLaaS testing applied to an IoT computing system with networked systems may include testing interactions (e.g., non-linear interactions) of the elements (as compared to testing the number of the individual elements). For example, in deciding the speed of a vehicle, the speed of the said vehicle is set based on a number of factors comprising the speed of vehicles preceding and following in the same lane, as well as vehicles merging into the lane, and possibly vehicles in other lanes. Ideally, the speed advisory system considered in the example herein may consider all such factors, comprising the speed of all vehicles on a road, and set the speed for all vehicles on the road, jointly. As the connectivity and automation of the individual elements of a networked system improves, it may be possible to monitor and simulate the interactions, whereby the simulation accuracy increases.

To elaborate upon the speed advisory example, assume that in a selected geographical region such as, for example, region "A," a speed-advisory system is in operation, with input being the positions of some number of actual vehicles on the street. Traditionally, such systems are tested only in simulations, and then deployed to all users, such that all users have to react to a simulated test. In HiLaaS testing, the HiLaaS testing system may be deployed for one or more of the road users, and may simulate the behavior of other road users, while assimilating sensor data, such as the ramp metering data. One or more rules in the simulation environment may be defined such as, for example, the speed of a simulated vehicle cannot exceed the speed of the actual vehicle, whose position is input and assimilated, unless taking-over is simulated. Other rules, such as the compliance of the road users with the advised speed, may be an approximation. Still, the HiLaaS testing allows for estimating the properties of the network system, such as traffic density and its evolution over time, by considering the various interactions between both real and simulated.

As an additional example, in power systems, a drop in reported wind-speed may trigger a drop in demand for active power as large users of electric power (such as a company or organization) expect the price of power to increase. Such interaction can also be modelled in the HiLaaS testing. It should be noted that a large number of actual entities may be preferable in the HiLaaS test. In the transportation applications, if only a small number of actual vehicles provide their positions and a small number of actual vehicles are "adversarial", e.g., driving very slowly so as to intentionally try to create an impression of traffic congestion, the speed advisory system may provide sub-optimal recommendations. As a result, many drivers may re-route their trips, even if this would not be strictly necessary, in a large-scale deployment.

In order to test a large-scale networked system, high-performance computing (e.g., hundreds and/or thousands of compute-nodes) may be required for a selected period of time (e.g., short periods of time). While it may not be economical to purchase the high-performance hardware for the purposes of the test, HiLaaS testing provides access to the high-performance hardware "as a service" (e.g., as a service on demand), whereby the high-performance computing facilities are shared in a cloud computing environment.

In one embodiment, the mechanisms of the present invention may use various simulation parameters (e.g., "inputs") into a simulation environment for the HiLaaS testing. The simulation parameters (e.g., "inputs") may include, but are not limited to, static data related to the network system (e.g., description and details of a power system, description and details of a transportation network, etc.), data for a selected time period (e.g., "per-day data" such as unit commitment schedule, a transportation timetable/schedule, calendaring data, etc.), real-time streamed data from the one or more entities and an environment associated with a networked system (e.g., a load of interruptible loads, positions of test vehicles), and information relating to the one or more entities (e.g., a type of simulation engine for HiLaaS testing, configuration of the HiLaaS testing, behavior of entities, or other user defined information).

Using the various simulation parameters or "inputs," the HiLaaS testing may estimate a number of entities to simulate and/or estimate interactions among the simulated entities, the real entities (e.g., non-simulated entities), and the environment of the networked system. The output of the HiLaaS testing of the networked system may provide real-time streamed data for the simulated entities and a forecast of the same data for the simulated entities for possible use in other applications. It should be noted that behavior of a network system may be forecasted for one or more reasons or motivations such as, for example, so as to set prices of services offered in real-time based on the forecast (e.g., forecast behavior of the network system to offer HiLaaS services at a selected price). The output of the HiLaaS testing of the networked system may also estimate one or more properties of the networked system based on the simulated entities and forecasts thereof.

Other examples of various aspects of the illustrated embodiments, and corresponding benefits, will be described further herein.

It is understood in advance that although this disclosure includes a detailed description on cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment.

Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service.

At the heart of cloud computing is an infrastructure comprising a network of interconnected nodes.

Referring now to <FIG>, a schematic of an example of a cloud computing node is shown. Cloud computing node <NUM> is only one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention described herein. Regardless, cloud computing node <NUM> is capable of being implemented and/or performing any of the functionality set forth hereinabove.

In cloud computing node <NUM> there is a computer system/server <NUM>, which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with computer system/server <NUM> include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

Computer system/server <NUM> may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system/server <NUM> may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

As shown in <FIG>, computer system/server <NUM> in cloud computing node <NUM> is shown in the form of a general-purpose computing device. The components of computer system/server <NUM> may include, but are not limited to, one or more processors or processing units <NUM>, a system memory <NUM>, and a bus <NUM> that couples various system components including system memory <NUM> to processor <NUM>.

Bus <NUM> represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnects (PCI) bus.

Computer system/server <NUM> typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server <NUM>, and it includes both volatile and non-volatile media, removable and non-removable media.

System memory <NUM> can include computer system readable media in the form of volatile memory, such as random access memory (RAM) <NUM> and/or cache memory <NUM>. Computer system/server <NUM> may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system <NUM> can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus <NUM> by one or more data media interfaces. As will be further depicted and described below, system memory <NUM> may include at least one program product having a set (e.g., at least one) of program modules that are configured to vehiclery out the functions of embodiments of the invention.

Program/utility <NUM>, having a set (at least one) of program modules <NUM>, may be stored in system memory <NUM> by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules <NUM> generally carry out the functions and/or methodologies of embodiments of the invention as described herein.

Computer system/server <NUM> may also communicate with one or more external devices <NUM> such as a keyboard, a pointing device, a display <NUM>, etc.; one or more devices that enable a user to interact with computer system/server <NUM>; and/or any devices (e.g., network vehicles, modem, etc.) that enable computer system/server <NUM> to communicate with one or more other computing devices. Such communication can occur via Input/Output (I/O) interfaces <NUM>. Still yet, computer system/server <NUM> can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) via network adapter <NUM>. As depicted, network adapter <NUM> communicates with the other components of computer system/server <NUM> via bus <NUM>. It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computer system/server <NUM>. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc..

As shown, cloud computing environment <NUM> comprises one or more cloud computing nodes <NUM> with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone 54A, desktop computer 54B, laptop computer 54C, and/or automobile computer system 54N may communicate.

Device layer <NUM> includes physical and/or virtual devices, embedded with and/or standalone electronics, sensors, actuators, and other objects to perform various tasks in a cloud computing environment <NUM>. Each of the devices in the device layer <NUM> incorporates networking capability to other functional abstraction layers such that information obtained from the devices may be provided thereto, and/or information from the other abstraction layers may be provided to the devices. In one embodiment, the various devices inclusive of the device layer <NUM> may incorporate a network of entities collectively known as the "internet of things" (IoT). Such a network of entities allows for intercommunication, collection, and dissemination of data to accomplish a great variety of purposes, as one of ordinary skill in the art will appreciate.

Device layer <NUM> as shown includes sensor <NUM>, actuator <NUM>, "learning" thermostat <NUM> with integrated processing, sensor, and networking electronics, camera <NUM>, controllable household outlet/receptacle <NUM>, and controllable electrical switch <NUM> as shown. Other possible devices may include, but are not limited to various additional sensor devices, networking devices, electronics devices (such as a remote control device), additional actuator devices, so called "smart" appliances such as a refrigerator or washer/dryer, and a wide variety of other possible interconnected objects.

Metering and Pricing <NUM> provides cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources may comprise application software licenses. Service Level Agreement (SLA) planning and fulfillment <NUM> provides pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.

Workloads layer <NUM> provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation <NUM>; software development and lifecycle management <NUM>; virtual classroom education delivery <NUM>; data analytics processing <NUM>; transaction processing <NUM>; and, in the context of the illustrated embodiments of the present invention, various workloads and functions <NUM> for testing embedded systems and application using HiLaaS. In addition, workloads and functions <NUM> for testing embedded systems and application using HiLaaS may include such operations as data analytics, data analysis, and as will be further described, notification functionality. One of ordinary skill in the art will appreciate that the workloads and functions <NUM> for testing embedded systems and application using HiLaaS may also work in conjunction with other portions of the various abstractions layers, such as those in hardware and software <NUM>, virtualization <NUM>, management <NUM>, and other workloads <NUM> (such as data analytics processing <NUM>, for example) to accomplish the various purposes of the illustrated embodiments of the present invention.

As previously mentioned, the mechanisms of the illustrated embodiments provide novel approaches to test embedded systems and application using HiLaaS in a computing environment. Simulation parameters may be received and/or defined to be tested in a simulation environment using one or more real entities and one or more simulated entities. In the simulated environment, the one or more simulated entities of one or more entities and/or the real entities in a networked system may be tested in real-time according to the simulation parameters. One or more properties of the one or more entities, the network system, or combination thereof may be estimated based on the testing of the one or more simulated entities.

In one embodiment, in the simulated environment, one or more simulated entities and one or more real entities in a networked system may be tested in real-time according to received control parameters, for a selected price (e.g., a price within a selected range, or below a pricing threshold). The price is estimated by the system, based on other parameters, and offered to the user to accept or reject. Alternatively, the user may specify the price, the system estimates control parameters, and the user can accept or reject the control parameters. One or more properties of the one or more entities, the network system, or combination thereof may be estimated based on the testing of the one or more simulated entities, when the price and control parameters are accepted.

In one embodiment, the one or more entities may interact with each other and the environment associated with the networked system. The number of the simulated entities in the simulation environment may be adjusted or modified over a selected period of time. That is, the number of simulated entities over the selected period of time does not comprise samples from an identical distribution, but is influenced, selected, chosen, and/or even determined by real-time information from real entities (e.g., real-time streamed data from the real entities), as well as influenced, selected, chosen, and/or even determined by historical data and schedule data for a given day. Consider, for example, transportation applications. Traditionally, it may be expected that the number of vehicles on the road (e.g., a network or "road network") is sampled from a model of demand, which is either static or specific to each hour of the week. This number of vehicles may be adjusted, however, considering the input from the actual vehicles, such as the speed of the vehicles, and input from third-party sensors such as, for example, induction loops of a traffic control system or sensors measuring the amount of snow-fall. This may produce more accurate simulation.

In addition, the mechanisms of the illustrated embodiments may incorporate real-time feedback, which may be obtained from one or more users, the network system, the simulated entities and/or the real entities of the network system. For example, the feedback obtained may include real-time streamed data of the one or more real entities. In this way, the feedback information may also be used to vary, update, alter, and/or modify the various simulation parameters.

Additionally, the mechanisms of the illustrated embodiments provide a flexible "shared" service to serve multiple users, that may input data from multiple real elements of a networked system under test, sharing the data completely, partially, or not at all even within one application domain. Also, the mechanisms of the illustrated embodiments can serve a wide range of applications (for traffic simulations, e.g., acceleration advice, speed recommendation, routing advice, etc.) where each application relies on a particular combination of real, emulated, and/or simulated data and data exchanges. The real elements of the networked system under test can have full, partial, or no access to the virtual environmental data simulated in the simulation environment.

Turning now to <FIG>, a diagram depicting an exemplary system for traffic simulation testing in an loT network environment. <FIG> depicts a diagram of exemplary computing, data processing, sensory, testing embedded systems and applications using HiLaaS, and other components <NUM> shown in the context of a moving vehicle <NUM>, in which various embodiments of the present invention may be implemented. The various components <NUM> may work together in concert for testing embedded systems and application in an loT network environment using HiLaaS.

As illustrated in <FIG>, an example of simulating traffic of one or more vehicles <NUM> (e.g., loT network entities) in a system under test <NUM> (e.g., simulation environment) using a test system <NUM> (e.g., a simulation engine) is depicted. That is, the one or more vehicles <NUM> may be tested in the system under test <NUM>. However, the system under test <NUM> using the HiLaaS <NUM> (e.g., a HiLaaS interface <NUM>) may be employed in a variety of contexts, scenarios, and/or other types of networked systems. As used herein, a "vehicle" may be an automobile, bicycle, hovercraft, scooter, bus, motorcycle, boat, ship, aircraft, plane, helicopter, drone, off road vehicle, truck, tractor, and/or other device used for movement or transportation. Additionally, a "vehicle" may be a virtual vehicle that may be used by system under test <NUM> for simulating one or more "real" or actual vehicles for testing in the system under test <NUM>. In one embodiment, one or more users such as, for example, user <NUM> may also be in communication with the HiLaaS <NUM> (e.g., a HiLaaS interface <NUM>).

In one embodiment, the vehicles <NUM> may be only one type of vehicle (e.g., only automobiles). Alternatively, the vehicles <NUM> may be a combination of types of vehicles (e.g., automobiles, motorcycles, busses, scooters, cyclists on a bicycle, etc.). Each of the vehicles <NUM> may include one or more types of loT devices (e.g., loT sensor devices) such as, for example, fuel consumption sensors, cameras, radio frequency identification "RFID" readers, biometric sensors, wearable sensors, computers, handheld devices (e.g., Global Positioning System "GPS" device or step counters), smartphones, and/or other sensor based devices.

In one embodiment, the system under test <NUM> may be a speed advisory system, which sets the speed for vehicles <NUM> based on <NUM>) positions of the vehicles <NUM>, <NUM>) real-time ramp metering data, and/or <NUM>) further observations of demand (e.g., observations from a base station of a Global System for Mobile Communications ("GSM") network). For example, the observations of demand may be a number of vehicles intending to move from point A to point B. For varying the number of vehicles intending to move from point A to point B, one or multiple origin-destination ("OD") matrices may be used. Given the observations of demand may be time-varying and also conditional, other representations may be used for the vehicles.

The test system <NUM> may be used to test and validate one or more properties of the system under test <NUM>. The test system <NUM> may be one or more simulator engines such as, for example, a micro-simulator of vehicular-traffic (e.g., Simulation of Urban Mobility ("SUMO"), a macroscopic simulator of vehicular traffic (e.g., based on a network of queues), a multi-model simulator ("MOVSIM"), a pedestrian simulator, and/or a cyclist simulator.

Moreover, the test system <NUM> uses the HiLaaS <NUM> to test a network system such as, for example, the system under test <NUM> in a simulation environment or "virtual" environment. The HiLaaS <NUM> may send data to the test system <NUM> and/or receive data from the test system <NUM>. More specifically, the one or more vehicles <NUM> may provide real-time streamed data to the test system <NUM> via the HiLaaS <NUM>.

As illustrated, the user <NUM> may send one or more control parameters to the HiLaaS <NUM> (e.g., the user may set the control parameters for the HiLaaS <NUM>). For example, the user <NUM> may send via the HiLaaS interface <NUM> to the test system <NUM>, one or more control parameters. The HiLaaS <NUM> may provide to the user pricing information for performing the testing. The user <NUM> may authorize and/or approve the price information.

Moreover, the one or more vehicles <NUM> may provide the HiLaaS <NUM> with a status update ("status()") of information relating to the one or more vehicles <NUM> (e.g., position location, speed data, turn signal information, etc.). In turn, the test system <NUM> may receive traffic information (e.g., using a "get traffic info()" function or "get environment info()" function operation) from the HiLaaS <NUM> such as, for example, receiving data from a selected number of the one or more vehicles <NUM>. The HiLaaS <NUM> may provide data relating to the real or "actual" vehicles of the one or more vehicles <NUM>. Thus, the test system <NUM> may provide an "illusion" to a user by simulating the actual vehicles <NUM> in an actual environment.

Additionally, using one or more simulation parameters, the system under test <NUM> may indicate (as the output of the test) the behavior of one or more vehicles <NUM> such as, for example, the speed that each vehicle <NUM> should be traveling at each point in a trajectory, the direction each vehicle should be driven, or other output of the simulation parameters. Alternatively, the one or more vehicles <NUM> may retrieve from the system under test <NUM> specific information (e.g., "GetSomeService()" function) relating to the simulation environment. In one embodiment, by way of example only, the test system <NUM> comprises a traffic simulator in a transportation setting. The system under test <NUM> may be a speed advisory system. Thus, the test system <NUM> may be a simulator and the system under test <NUM> may be a system that is to be tested.

In operation, the test system <NUM> may consider a set of "scenarios" (which may include simulation parameters and rules) such as, for example, high-level, abstract situations, to validate some "safety properties" across all of the scenarios. For example, one scenario may test when two lanes merge into one following an incident. Each safety property can be described using a modal and temporal logic, which allows you to provide to HiLaaS <NUM> text language such as, for example, "no vehicles at any trajectory point may get closer than <NUM> centimeters of each other", while a user or entity may be reasoning about simulation parameters and rules including the vehicles <NUM> (e.g., velocity, acceleration, braking, etc.) and the system under test <NUM> (e.g., the speed advisory system).

In order to "instantiate" and make specific the "scenario", the HiLaaS <NUM> enables simulating the actual movements of each individual vehicle of the vehicles <NUM> for the system under test <NUM> "as if" <NUM>) the speed advisory system was in place in the loT network and <NUM>) the "scenario" was in actual operation in the loT network. Based on data about GPS-trajectories from a selected number of the vehicles <NUM> (whose drivers are made aware of the simulation scenario via audio or visual means such as, for example, projection on the wind shield), the test system <NUM> derives trajectories of other vehicles <NUM> (which are either completely virtual, or correspond to known arrivals to a road segment recorded by induction loops (e.g., a means of counting vehicles such as, for example, inductive-loop traffic detectors), and/or without further available data on the trajectory).

As a further explanation of the HiLaaS <NUM> and the test system <NUM>, the HiLaaS <NUM> and the test system <NUM> may work in conjunction with each other, but the HiLaaS <NUM> may be agnostic of the system under test <NUM>. The HiLaaS <NUM> may be a service that allows simulations to be run on a cloud computing system and real entities to be plugged or associate to the simulations of a simulation environment. The HiLaaS <NUM> may be application-independent. To test a specific application or service, a set of test cases and scenarios may be executed and evaluated, which may be the role of the test system <NUM>. The test system <NUM> may be a system that runs specific scenarios (using the HiLaaS <NUM>) related to an application and measures the success of the application in those scenarios.

The HiLaaS <NUM> may be related to a unit testing framework that allows tests to be executed, but is agnostic of the system under test, and the test system <NUM> may be related to the unit tests, which are specific for a given system under test and properties to be tested ("application"). For example, assume that a speed advisory system is tested for vehicles. The objective of the speed advisory system is to suggest optimum speeds for vehicles with the purpose to reduce pollution, while maintaining the traffic in "good conditions" for one or more reasons. To evaluate the impact of the speed advisory system on road safety, evaluate how well the speed advisory system is performing in different conditions, and/or to study the effect of the speed advisory algorithm parameters on the pollution and traffic congestion, the test system <NUM> may be provided, which has a set of predefined scenarios and parameters. The test system <NUM> may run simulations on the HiLaaS <NUM>, possibly with one or more actual (or real) vehicles connected to the simulations, and the test system <NUM> monitors the results of the speed advisory system and effects on the overall traffic and pollution. The test system <NUM> can also analyze data coming from the real vehicles and drivers reflecting the satisfaction of drivers or the effect of the speed advisory on their behavior or vice-versa.

Turning now to <FIG>, an additional example of simulating traffic of one or more vehicles <NUM> (e.g., loT network entities) using a test system <NUM> (e.g., a simulation engine) is depicted. That is, <FIG> depicts exemplary computing, data processing, sensory, testing embedded systems and application using HiLaaS, and other components <NUM> shown in the context of the one or more vehicles in a traffic simulation. As will be seen, one or more embodiments of <FIG> may also be included with and/or implemented with <FIG>. For example, components <NUM> may be included in and/or associated with computer system/server <NUM> of <FIG>, incorporating one or more processing unit(s) <NUM> to perform various computational, data processing and other functionality in accordance with various embodiments of the present invention. The various components <NUM> may work together in concert for testing embedded systems and application in an loT network environment using HiLaaS.

As illustrated, the components <NUM> include a user <NUM>, the one or more vehicles <NUM> of <FIG>, the HiLaaS interface <NUM> (e.g., a HiLaaS Restful Application programming interface "API"), and one or more traffic/network simulator(s) such as, for example the test system <NUM> of <FIG>. In one embodiment, operations described in <FIG> between the user <NUM> and the HiLaaS <NUM> may also occur between the user <NUM> and the HiLaaS interface <NUM> (e.g., set control parameters, estimate/set pricing of the HiLaaS, and approve the pricing).

In operation, the user <NUM> may send one or more queries to the HiLaaS interface <NUM>. For example, the user <NUM> may send via the HiLaaS interface <NUM> to the test system <NUM>, a start simulation operation (e.g., a "StartSimulation" function), request a status update on the simulation (e.g., a "GetSimulationStatus" function), update the simulation (e.g., a "UpdateSimulation" function), and/or send a stop simulation command (e.g., a "StopSimulation" function).

The one or more vehicles <NUM> may also provide real-time streamed data to the HiLaaS interface <NUM>. For example, the one or more vehicles <NUM> may send via the HiLaaS interface <NUM> to the test system <NUM> data to check in the one or more vehicles <NUM> (e.g., a "CheckinVehicle" function). Using the HiLaaS interface <NUM>, the one or more vehicles <NUM> may also provide real-time streamed data that updates data of the one or more vehicles <NUM> (e.g., a "UpdateVehicleData" function), request a status update on the simulation (e.g., a "GetSimulationStatus" function), and/or send real-time streamed data to check out the vehicle (e.g., "CheckOutVehicle" function such as, for example, a function that stops considering the actual vehicle in the test system <NUM> and/or removes the vehicle from consideration from the test system <NUM>).

In the context of the mechanisms of the illustrated embodiments, and as previously described using <FIG>, the follow examples are provided as sample use cases. It should be noted that other use cases may also be applicable.

In a first use case, assume a user desires to analyze the driving behavior of drivers in response to different types of parking real-time information and associated consequences over the state of current traffic conditions. The user may register for the service (e.g., HiLaaS service) and may be provided credentials. The user may be provided access to a Rest-based API (or other type) to use the HiLaaS interface. The user may create a simulation and may define one or more simulation parameters such as, for example, defining a specific scenario via configuration files (e.g., network selection, traffic state, number of parking spots, parking demand, etc.). The user may check in one or multiple real vehicles in the simulation. The user can update the vehicle data into the simulation, for instance, at each GPS position update. The user may update (in real-time or "on the fly") the simulation (e.g., changing the simulated vehicle routes "on the fly"), which may require an update of the configuration file(s). The user can retrieve the simulation status and has access to all the simulated vehicles' data. The user can retrieve the status of any vehicle. The user may also use an analytics service which makes use of the retrieved simulation and vehicle status and provides advice to the real driven vehicle(s). The reaction of the real vehicle(s) to the given recommendations may be stored (as historical data) for later analysis. The user checks out the vehicles and stops the simulations (e.g., the vehicles may be removed from consideration by the test system <NUM>).

In a second use case, assume a user desires or intends to analyze a response of pedestrians to commercial routing advice in busy shopping streets, touristic cities, etc. so as to optimize customer satisfaction, a number of items purchased, comfort of walking, etc. The user may register for the service (e.g., HiLaaS service) and may be provided credentials. The user may be provided access to a Rest-based API (or other type) to use the HiLaaS interface. The user may create a simulation and may define specific scenarios via configuration files (e.g., area selection, street load, capacities of shopping stores, etc.). The user may check in one or multiple pedestrians in the simulation (e.g., pedestrians with GPS loggers). It should be noted that the cost of equipping pedestrians is low (e.g., an application may be used on a computing device that a user may download to the computing device). The user can update the pedestrian data into the (micro-)simulation, for instance, at each GPS position update. The user may update (in real-time or "on the fly") the simulation such as, for example, changing the simulated pedestrian routes on the fly, which may require an update of the configuration file(s). The user may update (in real-time or "on the fly") the simulation (e.g., changing the simulated vehicle routes "on the fly") and have access to all the simulated pedestrian data. The user can retrieve any vehicle status in particular, e.g., information about specific vehicle types. The user may also use an analytics service which makes use of the retrieved simulation and pedestrian status and gives advice to the equipped pedestrians. The reaction of the pedestrians to the given recommendations (and their impact on the pedestrian traffic) may be analyzed. The user may also use an analytics service. The user may check out the pedestrians from the simulation and stop the simulation.

In a third use case, assume a user desires to investigate a cyclist's behavior in response to routing advice such as, for example, pollution avoidance, or to optimize covered areas in the event of adverse weather conditions. The user may register for the service (e.g., HiLaaS service) and may be provided credentials. The user may be provided access to a Rest-based API (or other type) to use the HiLaaS interface. The user may create a simulation and may define one or more simulation parameters such as, for example, defining a specific scenario via configuration files (e.g., area selection, origin-destination matrices or other means of modelling the demand for transportation, traffic state, weather conditions, etc.). The user may check in one or multiple cyclists in the simulation such as, for example, cyclists with GPS loggers. It should be noted that the cost of equipping cyclists is low, such as, for example, by providing the cyclists with an application installed onto a mobile device (e.g., smartphone or other computing device), considering the wide spread of mobile phones capable of running third-party applications. The user can update the cyclists' data into the (micro-)simulation, for instance, at each GPS position update. The user may update, in real-time or "on the fly," the simulation such as, for example, changing the simulated cyclist's routes or weather conditions on the fly. The user may retrieve the simulation status and has access to all the simulated cyclist's data, as well as pollutant concentration data generated by vehicle traffic on the cyclist's routes. The user may retrieve a cyclist status (e.g., the cyclist status of those corresponding to a cyclist type). The user may also use an analytics service, which makes use of the retrieved simulation data and cyclist's status and gives advice to the real cyclists. The reaction of the cyclists to a given recommendation may be analyzed. The user is able to check out the cyclists from the simulation and stop the simulation.

Turning now to <FIG>, a method <NUM> for testing embedded systems and application in an Internet of Things (IoT) network environment using Hardware-in-the-Loop as a Service (HiLaaS) by a processor is depicted, in which various aspects of the illustrated embodiments may be implemented. The functionality <NUM> may be implemented as a method executed as instructions on a machine, where the instructions are included on at least one computer readable medium or one non-transitory machine-readable storage medium. The functionality <NUM> may start in block <NUM>. Control parameters and information relating to a simulation environment may be received, as in block <NUM>. In the simulated environment, one or more simulated entities of one or more entities in a networked system may be tested in real-time according to the simulation parameters, as in block <NUM>. One or more properties of the one or more entities, the network system, or combination thereof may be estimated based on the testing of the one or more simulated entities, as in block <NUM>. The functionality <NUM> may end, as in block <NUM>.

Turning now to <FIG>, an additional method <NUM> for testing embedded systems and application in an Internet of Things (IoT) network environment using Hardware-in-the-Loop as a Service (HiLaaS) by a processor is depicted. The functionality <NUM> may be implemented as a method executed as instructions on a machine, where the instructions are included on at least one computer readable medium or one non-transitory machine-readable storage medium. The functionality <NUM> may start in block <NUM>. For testing simulation parameters, static data related to the network system, data for a selected time period, real-time streamed data from the one or more entities and an external system associated with a networked system, and information relating to the one or more entities may be used (and/or received as part of the simulation parameters), as in block <NUM>. One or more control parameters (e.g., confidence intervals of an estimate of one or more properties of the networked system) may be set, as in block <NUM>. A number of the one or more entities to simulate over a selected period of time may be estimated and/or a number of interactions between the one or more entities and the one or more simulated entities, and an external system associated with the networked system may be estimated, as in block <NUM>. A TaaS may be approved (e.g., a price is estimated for the testing and the prices for the testing may be approved), as in block <NUM>. Real-time streamed data for the one or more simulated entities may be provided, as in block <NUM>. One or more properties of the network system may be estimated based on the one or more simulated entities, as in block <NUM>. The functionality <NUM> may end, as in block <NUM>.

In one embodiment, in conjunction with and/or as part of at least one block of <FIG>, the operations of methods <NUM> or <NUM> may include each of the following. The operations of methods <NUM> or <NUM> may receive data related to the network system, receive scheduling data for a selected time period, receive real-time streamed data from the one or more entities, and/or receive information relating to a type of simulator to use for the testing. A HiLaaS interface may be used and/or accessed by the one or more entities, a user, or a combination thereof for accessing the simulated environment.

A number of the one or more entities may be estimated to simulate over a selected period of time. A number of interactions between the one or more entities, the one or more simulated entities, and an environment associated with the networked system may also be estimated.

The operations of methods <NUM> or <NUM> may provide real-time streamed data for the one or more entities based on the testing of the one or more simulated entities. The operations of methods <NUM> or <NUM> may simulate the one or more entities according to the simulation parameters, and/or update, in real-time, the simulation parameters during the simulating. Additionally, the operations of methods <NUM> or <NUM> may synchronize one or more operations of the one or more entities and the one or more simulated entities in the simulation environment.

Turning now to <FIG>, a method <NUM> for testing embedded systems and application in an Internet of Things (IoT) network environment using a HiLaaS by a processor is depicted, in which various aspects of the illustrated embodiments may be implemented. The functionality <NUM> may be implemented as a method executed as instructions on a machine, where the instructions are included on at least one computer readable medium or one non-transitory machine-readable storage medium. The functionality <NUM> may start in block <NUM>. A description of a simulation environment and price may be received, as in block <NUM>. One or more control parameters of a test may be estimated according to the price and the description of the simulation environment, as in block <NUM>. One or more entities in a network system, one or more simulated entities, or a combination thereof may be tested according to the one or more control parameters upon acceptance of the one or more control parameters, as in block <NUM>. One or more properties of the one or more entities, the network system, or combination thereof may be estimated based on the testing of the one or more simulated entities, as in block <NUM>. The functionality <NUM> may end, as in block <NUM>.

Turning now to <FIG>, a method <NUM> for testing embedded systems and application in an Internet of Things (IoT) network environment using a HiLaaS by a processor is depicted. The functionality <NUM> may be implemented as a method executed as instructions on a machine, where the instructions are included on at least one computer readable medium or one non-transitory machine-readable storage medium. The functionality <NUM> may start in block <NUM>. One or more control parameters and information relating to a simulation environment may be received, as in block <NUM>. In the simulation environment, one or more entities associated with a network system, one or more simulated entities associated with the network system, or a combination thereof may be tested, as a testing as a service ("TaaS" such as, for example, the HiLaaS), according to the one or more control parameters, as in block <NUM>. One or more properties of the one or more entities, the network system, or combination thereof may be estimated based on the testing of the one or more simulated entities, as in block <NUM>. The functionality <NUM> may end, as in block <NUM>.

In one embodiment, in conjunction with and/or as part of at least one block of <FIG>, the operations of methods <NUM> or <NUM> may include each of the following. The methods <NUM> or <NUM> may estimate a price for testing in the simulation environment according to the one or more control parameters; receive an approval of the estimated price for the testing; and/or test the one or more entities, the one or more simulated entities, or a combination thereof. Moreover, the methods <NUM> or <NUM> may receive a maximum threshold price, a range of pricing, or a combination thereof for testing in the simulation environment; adjust one or more control parameters according to the maximum threshold price, the range of pricing, or a combination thereof for testing in the simulation environment; receive an approval of the adjusted control; and/or test the one or more entities, the one or more simulated entities, or a combination thereof.

Pursuant to receiving the one or more control parameters and the information relating to the simulation environment, the methods <NUM> or <NUM> may receive a description of the simulation environment, receive a description of the network system, wherein the description includes at least a topology of the network system, receive historical data related to operations of the network system, receive scheduling data for a selected time period of the operations of the network system, receive real-time streamed data from the one or more entities, receive real-time streamed data from one or more third-parties, wherein the streamed data may include weather monitoring data, pollution monitoring data, satellite imagery data, or aerial imagery data, or a combination thereof (and/or other data provided by a third party), receive a selected time to perform the testing and provide testing results, receive one or more conditions for the testing the one or more simulated entities, receive one or more conditions for the testing the one or more entities of the system under test, and/or receive a measure of accuracy of the testing.

The methods <NUM> or <NUM> may estimate a number of the one or more entities to simulate over a selected period of time according to the one or more control parameters, estimate a number of interactions between the one or more entities, the one or more simulated entities, and an environment associated with the networked system according to the one or more control parameters, and/or estimate one or more computing systems needed to simulate the one or more entities and the number of interactions over a selected time period according to the one or more control parameters.

The methods <NUM> or <NUM> may simulate the one or more entities according to the control parameters, update the simulation environment in real-time during the simulating, wherein the updating includes at least historical data of operations of the network system, and/or provide real-time streamed data for the one or more simulated entities as output of testing of the one or more simulated entities. The methods <NUM> or <NUM> may provide an interface for remote access by the one or more entities, a user, or a combination thereof for accessing the simulated environment.

The methods <NUM> or <NUM> may receive a description of a simulation environment and price; estimate the one or more control parameters of a test according to the price and the description of a simulation environment; test the one or more entities associated, the one or more simulated entities, or a combination thereof according to the control parameters upon acceptance of the one or more control parameters; and/or estimate the one or more properties of the one or more entities, the network system, or combination thereof based on the testing of the one or more simulated entities.

The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out embodiments of the present invention.

In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform embodiments of the present invention.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowcharts and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowcharts and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowcharts and/or block diagram block or blocks.

Claim 1:
A method, by a processor, for testing networked systems in an Internet of Things, loT, environment, comprising:
receiving one or more control parameters and information relating to a simulation environment (<NUM>);
testing, as a Testing as a Service, TaaS, in the simulation environment, one or more real entities associated with a network system and one or more simulated entities associated with the network system according to the one or more control parameters (<NUM>), wherein the number of simulated entities under test is adjusted in real-time according to real-time information received from the one or more real entities; and
estimating one or more properties of the one or more entities, the network system, or combination thereof based on the testing of the one or more simulated entities (<NUM>),
wherein real-time data from a real entity affects the behaviour of a simulated entity.