Patent Description:
Machine-to-machine (M2M) technologies allow devices to communicate more directly with each other using wired and wireless communications systems. M2M technologies enable further realization of the Internet of Things (IoT), a system of uniquely identifiable objects and virtual representations of such objects that communicate with each other and over a network, such as the Internet. IoT may facilitate communications with even mundane everyday objects, such as products in a grocery store or appliances in a home, and thereby reduce costs and waste by improving knowledge of such objects. For example, stores may maintain very precise inventory data by being able to communicate with, or obtain data from, objects that may be in inventory or may have been sold.

IoT entities and systems are highly diverse and may have a wide variety of characteristics, applications, and functions. As will be appreciated, this diversity of entities generates and receives a wide variety of data. Data collection may take place at a "front-end" entity, such as a sensor and sent to "back-end" entities, such as gateways or networks. Collected data may be stored and processed at similar back-end devices, and provided to users and applications anywhere in an IoT system. As many IoT entities may be relatively simple devices, such devices may generate data streams of raw data with little or no contextual information. Having such information may increase the performance and efficiency of IoT systems.

<CIT> relates to a system for synchronous delivery of annotated multimedia streams.

The embodiments set forth herein may be described in terms of a representational state transfer (REST) architecture, with components and entities described conforming to the constraints of a REST architecture (RESTful architecture). A RESTful architecture is described in terms of the constraints applied to components, entities, connecters, and data elements used in the architecture rather than in terms of physical component implementation or communications protocols used. Thus, the roles and functions of the components, entities, connecters, and data elements will be described. In a RESTful architecture, representations of uniquely addressable resources may be transferred between entities. One skilled in the art will recognize that implementations of the instant embodiments may vary while remaining within the scope of the present disclosure. One skilled in the art will also recognize that the disclosed exemplary embodiments, while sometimes described herein in reference to the European Telecommunications Standards Institute (ETSI) M2M architecture, are not limited to implementations using the ETSI M2M architecture. The disclosed embodiments may be implemented in other architectures and systems that have connected entities, such as oneM2M and other M2M systems and architectures.

IoT systems may include many different kinds of sensors, each of which may produce different types of raw sensory data. This data may have very diverse characteristics depending on the type of sensor and sensor requirements. In one embodiment, "small data" that is several or tens of bytes may be generated by a sensor. For example, a temperature reading from a temperature sensor may be small data. To transmit each piece of small data separately may introduce excessive overhead in an IoT system and/or entities therein, especially when a large number of sensors may be involved and/or each sensor is continuously generating data. Data generated by a sensor continuously (i.e., the sensor generates a data stream) may have high tempo correlation. That is, the data may be unchanged for long periods of time, such as temperature data for a room. Such tempo-correlated data may be aggregated without losing key information. Note that a single reading may be considered a special data stream. Data generated by several sensors of the same type scattered about a physical region may have spatial correlation, where the multiple sensors generate very similar or highly correlated data. Thus, this type of data may also be aggregated without a loss of key information. Sensory data from different types of sensors may have application-level correlation, where their functions or services are related in some way. For example, data from a smoke sensor and a temperature sensor may be correlated with each other if both are triggered by a fire. In another example, data from a body sensor on an asthmatic patient may be correlated to sensory data from a smoke sensor or a fragrance sensor. Such application-level correlated sensory data may be leveraged based on common or similar characteristics to improve event prediction and observation accuracy.

Data may be captured, or observed, in various manners in an IoT system. In an embodiment, data may be observed, transmitted, or otherwise collected in response to detected events (may be referred to as "event-based data observation"). An event may be detected, or an automatic notification may be generated on an occurrence of an event, and in response data may be transmitted or collected. Any type of event is contemplated, such as meeting or passing a threshold (e.g., low-level temperature event passing a temperature threshold), occurrence of a detectable event, (e.g., high-level emergency event such as detection of a fire), or any other event that may be detected or generated by an IoT entity. In another embodiment, data may be collected by an IoT entity as needed (may be referred to as "query-based data observation"), where the IoT entity sends a request for the desired data to an entity that has access to the desired data. In still other embodiments, data may be collected continuously (may be referred to as "continuous data observation"), where data may be continuously generated and/or collected (i.e., generation and/or collection of a data stream). Any of these data observation methods may be used in combination with any other to implement "hybrid data observation", where an IoT entity may jointly exploit event-based, query-based, and/or continuous data observation. Note that the entities described herein that perform data collection and/or data annotation may simply be referred to as "annotation entities". These entities but may be any one or more entities in any network of communicatively connected entities, and each such entity may be implemented in one or more devices, systems, networks, etc. All such embodiments are contemplated as within the scope of the present disclosure.

As noted, there may be many diverse raw data streams generated in an IoT system. Each data stream may include a number of data items. For example, a data stream may be a series of continuous readings from a smart electricity meter. Multiple data items of a data stream may be treated as a data window, and thus a data stream may have multiple data windows.

Raw data steams may not provide much useful information or value until they are collected and analyzed. To efficiently analyze raw data streams, or to further analyze data streams that have already been processed, additional information may be added into a data stream that may be used for IoT data mining and analytics. Such information may, for instance, facilitate determining correlations between data streams, interactions among IoT things, interactions among people and IoT things, etc. Adding additional information to an existing data stream may be referred to as "data annotation. " An annotation added to a data stream may be information that provides additional semantics or contextual information to describe the data steam in some way. Such information may be referred to as "data annotation concepts" or "data annotation data. " In an embodiment, a data stream may be modified to indicate the surrounding situation, or context, in which the data stream was generated, the relevant people, activity, or IoT thing status at the time of generation of the data stream, an event that triggered the generation of the data stream is generated, a condition under which the data stream was generated, a location, time, type, and/or source of a data stream, and/or a correlation among different data streams. An important factor in reducing the performance impact of data annotation may be determining an appropriate time to annotate the data stream (e.g., during data collection or after data collection) and a granularity of the data annotation (e.g., annotation added to a data item, a data window, a data stream, multiple streams, etc., described in more detail herein).

Data annotation can be leveraged for many IoT applications to annotate various IoT data streams, such as energy data in smart home energy systems where smart meters are deployed to measure real-time energy consumption or temperature data from temperature sensors. Raw data (e.g., instantaneous smart meter readings or temperature readings) may not be enough to fully determine needed characteristics of a system. For example, appliance status, activities being performed in the home, time of day, or number of people in a home may provide additional data that may be used to determine, for example, power consumption or appropriate temperature settings. Thus, additional data that indicates, for example, number of people in a home, etc., may be added to data streams as annotations for improved data stream processing. For example, a home gateway that connects to a smart meter may add notations to a data stream generated by sensors in the home reflecting additional data. While collecting energy data from a sensor, a gateway may simultaneously analyze other data from other sources, such as other sensors, to determine the annotation concept, which may include such data as appliance status, number of people in a home, etc..

Annotations may be added to a single data item, a data window containing one or more data items, an entire data stream, and to multiple data streams. Exemplary, non-limiting representation <NUM> of multi-level IoT data annotation is illustrated in <FIG>, which shows data streams <NUM> and <NUM>, containing data items <NUM>-<NUM> and <NUM>-<NUM>, respectively. Item-level annotations <NUM>-<NUM> may be annotated to some of the individual data items in stream <NUM> while data items <NUM>-<NUM> may be annotated to some of the data items in stream <NUM>, as shown in <FIG>.

Window-level annotations <NUM> and <NUM> may be annotated to data windows <NUM> and <NUM>, respectively, while data window <NUM> of stream <NUM> may not be annotated, while data item <NUM> in data window <NUM> may be annotated. Similarly, window-level annotations <NUM> and <NUM> may be annotated to data windows <NUM> and <NUM>, respectively, while data window <NUM> of stream <NUM> may not be annotated, while data item <NUM> in data window <NUM> is annotated. The use of window-level annotations may allow the sharing of an annotation by multiple data items within a same data window, in some embodiments reducing overhead compared to item-level annotation.

Entire streams may also be annotated. As seen in <FIG>, data stream annotation <NUM> may be applied to data stream <NUM> while data stream annotation <NUM> may be applied to data stream <NUM>. The use of data stream-level annotation may allow the sharing of an annotation by all data items within a same data stream, in some embodiments even further reducing overhead compared to window-level annotation.

In an embodiment, annotations may be applied to multiple streams. As seen in <FIG>, cross-stream annotation <NUM> may be applied to both data stream <NUM> and data stream <NUM>. The use cross-stream annotation may allow the sharing of an annotation by data items within multiple data streams, in some embodiments even further reducing overhead compared to data stream-level annotation. In one such embodiment, as shown in <FIG>, cross-full-stream annotation may be applied to all data items within multiple data stream, for example, as annotation <NUM> is applied to all data items of data streams <NUM> and <NUM>.

Alternatively, referring now to <FIG> illustrating exemplary, non-limiting representation <NUM> of multi-level IoT data annotation, cross-partial-stream annotation may be employed, where annotation is applied across data streams, but not across each entire data stream of the multiple data streams. As shown in <FIG>, similar to <FIG>, item-level annotations <NUM>-<NUM> may be annotated to some of the individual data items in stream <NUM> while data items <NUM>-<NUM> may be annotated to some of the data items in stream <NUM>, window-level annotations <NUM>, <NUM>, <NUM>, and <NUM> may be annotated to data windows <NUM>, <NUM>, <NUM>, and <NUM>, respectively, while data windows <NUM> and <NUM> may not be annotated, and data stream annotation <NUM> may be applied to data stream <NUM> while data stream annotation <NUM> may be applied to data stream <NUM>. In this embodiment, cross-partial-stream annotation <NUM> may be applied to a subset of the data items of both of data streams <NUM> and <NUM>, while cross-partial-stream annotation <NUM> may be applied to another subset of the data items of both of data streams <NUM> and <NUM> as shown in <FIG>.

As noted, the disclosed multi-level data annotation provides multiple data annotation approaches to assist in the reduction of total overhead and increasing the efficiency of data annotation. Note that in the disclosed embodiments, multiple different annotations may be applied to a single data item, data window, data stream, or the same set or subset of multiple data streams.

In an embodiment, data annotation may be provided as an IoT service capability that may be referred to as "Data Annotation as a Service" (DAaaS). DAaaS may be integrated into an existing IoT service capability or service layer platform (e.g., ETSI M2M, oneM2M) or implemented as a standalone service capability to be leveraged by other service capabilities or common service entities. DAaaS may reside at an IoT device, an IoT gateway, and/or an IoT server. <FIG> illustrates the interaction of various functions and entities that may implement DAaaS. An IoT entity may perform data stream analysis <NUM> where data stream <NUM> (e.g., from a smart meter or a temperature sensor) is analyzed to determine context information associated with stream <NUM>. For example, context information associated with stream <NUM> may include data about the environment about the sensor generating stream <NUM>, such as the status of proximate things (e.g., appliances, people, activities, etc.), that may be used to automatically determine annotation concepts that may be applied to other data streams (e.g., utility consumption from smart meters, temperature data from temperature sensors). Data stream analysis function <NUM>, which may be implemented in one or more IoT entities, may determine the context data for input data stream <NUM> and the determined context data may be stored at annotation concept management <NUM>, which may be a database or other entity that may store data that may be used to annotate data streams.

In an embodiment, automatic data annotation (ADA) <NUM> may be a function implemented by one or more IoT entities that, upon detection of a condition, automatically trigger a data annotation process that annotates data streams. ADA <NUM> may involve interactions between data stream analysis function <NUM>, the internal concept base of annotation concept management function <NUM>, and data annotation process <NUM>. ADA enables automated data annotation during data collection. For example, a home gateway may collect data from one or more smart meters in a house. The gateway may also receive or obtain context data from one or more other sensors, such as motion sensors, sound sensors, light sensors, and temperature sensors, that may be deployed to monitor various statuses and activities in the house while collecting smart meter data. The gateway may analyze the context data received from the other sensors to obtain status and/or activity data that it may then use to annotate the smart meter data stream. The annotation concepts or context data may be raw data received from a sensor or data determined by the gateway or another IoT entity based on information received from one or more IoT applications. Annotation concept management <NUM>, having stored the context data received from data stream analysis function <NUM> or other annotation concepts or data, may provide such data to data annotation process function <NUM> that may annotate a data stream to generate annotated data stream <NUM>.

Alternatively, or in addition, external annotation requesters <NUM> may request that an annotation concept be stored at annotation concept management <NUM> in requested data annotation process (RDA) <NUM>. RDA <NUM> may be considered annotation on demand. In RDA <NUM>, data may be collected and stored in response to requests from annotation requestors <NUM> that may be any IoT devices, gateways, servers, applications, other DAaaSs, etc. The data collected or obtained in response to such requests may be provided to data annotation process function <NUM> by annotation concept management <NUM>. Data annotation process function <NUM> may in turn annotate a data stream with such data to generate annotated data stream <NUM>. Annotation enablement <NUM> may provide one or more interfaces for annotation requestors <NUM> to interact with DAaaS <NUM>. Alternatively, or in addition, requested data annotation process <NUM> may allow external concept base <NUM>, which may be any one or more IoT entities that may provide annotation concepts, to interact with DAaaS <NUM> and provide annotation concepts that may be used by annotation process <NUM> to annotate a data stream, generating annotated data stream <NUM>. Any entity that stores and/or provides annotation concepts or data may also be referred to as an "annotation concept entity" or an "annotation data entity".

Annotation requestors <NUM> may send one or more annotation requests to DAaaS <NUM> to trigger requested data annotation process <NUM>. Annotation requestors <NUM> may specify an external concept or other annotation data that may be stored in external concept base <NUM>, or may provide such data directly from annotation requesters <NUM> in the annotation request. Annotation requestors <NUM> may also, or instead, access and manipulate the internal concept base of annotation concept management function <NUM> via annotation enablement <NUM> interfaces.

Data annotation process <NUM> may operate in one or both of automatic data annotation process <NUM> and requested data annotation process <NUM>. Data annotation process <NUM> may receive annotation requests and/or triggers from data stream analysis <NUM> and/or annotation requesters <NUM> via annotation enablement <NUM>. Data annotation process <NUM> may determine an appropriate multi-level data annotation scheme. Data annotation process <NUM> may use both the internal concept base of annotation concept management <NUM> and external concept base <NUM> to determine and perform data annotation. Data annotation process <NUM> may also have an interface to the one or more data streams to be annotated.

Various embodiments of the present disclosure may employ various DAaaS architectures that may enable DAaaS as a service capability or common service function to interact with other IoT entities (e.g., nodes, services, etc.). <FIG> illustrates exemplary, non-limiting DAaaS architecture <NUM>, where a DAaaS resides in an IoT node (e.g., an IoT device, gateway, or server). Such DAaaSs may interact with other IoT Services, other DAaaSs, and one or more IoT semantic concept servers, such as IoT semantic concept server <NUM>. IoT semantic concept server <NUM> may be any IoT entity and any combination of multiple IoT entities and/or other devices and entities. IoT semantic concept server <NUM> may maintain annotation concepts and/or annotation data that may be accessed and used by a DAaaS.

In an embodiment, other IoT services <NUM> in IoT node <NUM>, which may or may not be receiving data stream <NUM>, may send data annotation request <NUM> to DAaaS <NUM> in IoT node <NUM>. In response, DAaaS <NUM> in IoT node <NUM> may apply annotations to the data stream indicated in request <NUM> (e.g., data stream <NUM> or data stream <NUM> obtained from a source not shown) and may send data annotation response <NUM> to other IoT services <NUM>. In order to determine the appropriate data annotations to apply, DAaaS <NUM> may query its internal concept base to determine appropriate annotation concepts and data. DAaaS <NUM> may also, or instead, access data and concepts from IoT semantic concept server <NUM> where annotation concepts and data may be stored at concept base <NUM>. DAaaS <NUM> may send concept access request <NUM> that may include data related to request <NUM> and/or IoT node <NUM>, to IoT semantic concept server <NUM> and subsequently receive concept access response <NUM> that may include annotation concepts and/or data.

DAaaS <NUM> may also, or instead, send data annotation request <NUM> to DAaaS <NUM> of IoT node <NUM>. For example, DAaaS <NUM> may determine that the entity that can perform annotation most efficiently may be DAaaS <NUM>. DAaaS <NUM> may add annotations to data streams received or maintained by IoT node <NUM> and indicated by annotation request <NUM>. IoT node <NUM> may then send data annotation response <NUM> to DAaaS <NUM>. As with DAaaS <NUM>, in order to determine the appropriate data annotations to apply, DAaaS <NUM> may query its internal concept base to determine appropriate annotation concepts and data. DAaaS <NUM> may also, or instead, access data and concepts from IoT semantic concept server <NUM> where annotation concepts and data may be stored at concept base <NUM>. DAaaS <NUM> may send concept access request <NUM>, that may include data related to request <NUM> and/or IoT node <NUM>, to IoT semantic concept server <NUM> and subsequently receive concept access response <NUM> that may include annotation concepts and/or data.

In another embodiment, other IoT services <NUM> in IoT node <NUM> may interact with IoT semantic concept server <NUM> to determine data annotation concepts and/or data. Other IoT services <NUM> may send concept access request <NUM> to IoT semantic concept server <NUM> requesting annotation data and/or concepts and receive concept access response <NUM> with the requested data. Other IoT services <NUM> may then include such data in its requests to DAaaSs, such as annotation request <NUM> sent to DAaaS <NUM>. This may allow DAaaS <NUM> to annotate one or more data streams, or components thereof, without requiring DAaaS <NUM> to query IoT semantic concept server <NUM>.

Note that any of the annotation requests and concept access requests described herein may be requests to retrieve existing annotation concepts and/or data, delete existing annotation concepts and/or data, update existing annotation concepts and/or data, and/or insert new annotation concepts and/or data into an IoT semantic concept server.

<FIG> illustrates the exemplary, non-limiting proxy-based DAaaS architecture <NUM>, where DAaaS <NUM> serves as a proxy between DAaaS <NUM> and DAaaS <NUM>. DAaaS <NUM> at IoT node <NUM>, which may or may not be receiving data stream <NUM>, may desire to request that one or more data streams in IoT node <NUM> be annotated, but it may not be able to reach DAaaS <NUM> directly. Therefore, DAaaS <NUM> may send data annotation request <NUM> to DAaaS <NUM> which may relay DAaaS <NUM>'s annotation request as annotation request <NUM> to DAaaS <NUM>. DAaaS <NUM> may receive annotation response <NUM> from DAaaS <NUM> indicating the success or failure of the annotation request. DAaaS <NUM> may forward the response to DAaaS <NUM> as annotation response <NUM>.

In some embodiments, DAaaS <NUM> may communicate in a format or other communications means that is not understood by DAaaS <NUM>. In such embodiments, DAaaS <NUM> may translate the request, concept, and/or data contained in annotation request <NUM> into another format or other form that DAaaS <NUM> can understand. DAaaS <NUM> may send such translated request, data, etc. to DAaaS <NUM> within annotation request <NUM>.

While serving as an annotation proxy, DAaaS <NUM> may combine and/or aggregate annotation requests from DAaaS <NUM> and/or other DAaaSs into a single annotation request, such as annotation request <NUM>, that may be sent to DAaaS <NUM>. For example, annotation request <NUM> from DAaaS <NUM> may request to annotate data items or data windows of data stream indicated in one or more other requests from other DAaaSs. The other DAaaSs may be requesting different annotations of a same data stream than those requested by DAaaS <NUM>. Only data stream may be in common among the requests, although in other embodiments, the annotations may be partially or wholly in common among the requests. In such an embodiment, DAaaS <NUM> may aggregate some or all of these requests into a single request and send that single annotation request to DAaaS <NUM> for application to the indicated data stream.

In various embodiments of the instant disclosure, procedures and corresponding messages used in DAaaS may be implemented in various ways. In one embodiment, DAaaS messages and procedures may be implemented as service layer primitives bound to various different lower layer protocols, such as the hypertext transfer protocol (HTTP), the simple object access protocol (SoAP), the constrained application protocol (CoAP), or any other protocol. Alternatively, DAaaS messages and procedures may be implemented using the payloads of application protocols such as HTTP and CoAP and others. In another embodiment, messages and procedures may be implemented as protocol header options of application protocols such as HTTP and CoAP and others. Alternatively, DAaaS messages and procedures may be implemented using device management protocols such as the open mobile alliance (OMA) device management (DM) protocol and others. In yet another alternative, DAaaS messages and procedures may be implemented using short message servive (SMS) in a wireless cellular network. All such embodiments are contemplated as within the scope of the present disclosure.

<FIG> shows exemplary, non-limiting signal flow <NUM> illustrating exemplary signals and procedures for automatic data annotation (ADA). ADA may include data annotation configuration phase <NUM> and data collection and annotation phase <NUM>, although phase <NUM> may be performed without first performing phase <NUM>.

In data annotation configuration phase <NUM>, IoT node <NUM>, which may be, in some embodiments, an IoT server, may send data annotation configuration request <NUM> to IoT node <NUM> that may be, in some embodiments, an IoT gateway. Data annotation configuration request <NUM> may include data indicating the data streams that IoT node <NUM> should collect and annotate, the data streams that IoT node <NUM> should collect and analyze to obtain annotation concepts and/or data, and/or the level and data stream on which IoT node <NUM> should perform data annotation (e.g., item-level annotation, window-level annotation, stream-level annotation, cross-stream annotation, cross-partial-stream annotation, etc.). For example, IoT node <NUM> may be requested to annotate a data stream transmitted from IoT node <NUM> based on annotation concepts and/or data to be determined from data streams emanating from IoT node <NUM> and IoT node <NUM>.

Upon receipt of data annotation configuration request <NUM>, IoT node <NUM> may determine that it needs to collect a data stream or data related to a data stream from IoT node <NUM> for analysis and determination of data annotation concepts and/or data in order to annotate the data stream transmitted from IoT node <NUM> according to the instructions in data annotation configuration request <NUM>. In response, IoT node <NUM> may send data request <NUM> to IoT node <NUM> instructing IoT node <NUM> to respond with the data specified in data request <NUM>. Data request <NUM> may enable IoT node <NUM> to report the requested data to IoT node <NUM> at data report <NUM>, described in further detail below. IoT node <NUM>, upon receipt of data request <NUM>, may send acknowledgement <NUM> to IoT node <NUM> acknowledging receipt of data request <NUM>.

Similarly, upon receipt of data annotation configuration request <NUM>, IoT node <NUM> may determine that it needs to collect a data stream or data related to a data stream from IoT node <NUM> for analysis and determination of data annotation concepts and/or data in order to annotate the data stream transmitted from IoT node <NUM> according to the instructions in data annotation configuration request <NUM>. In response, IoT node <NUM> may send data request <NUM> to IoT node <NUM> instructing IoT node <NUM> to respond with the data specified in data request <NUM>. Note that the requested data may be different for each of IoT node <NUM> and <NUM>. Data request <NUM> may enable IoT node <NUM> to report the requested data to IoT node <NUM> at data report <NUM>, described in further detail below. IoT node <NUM>, upon receipt of data request <NUM>, may send acknowledgement <NUM> to IoT node <NUM> acknowledging receipt of data request <NUM>.

Upon receiving the acknowledgements from IoT node <NUM> and IoT node <NUM>, IoT node <NUM> may send data annotation configuration response <NUM> to IoT node <NUM>.

In data collection and annotation phase <NUM>, IoT node <NUM> may send the data requested in data request <NUM> to IoT node <NUM> in data report <NUM>. Likewise, IoT node <NUM> may send the data requested in data request <NUM> to IoT node <NUM> in data report <NUM>. At block <NUM>, IoT node <NUM> may analyze the data streams and/or other data received from IoT node <NUM> and IoT node <NUM> to determine annotation concepts and/or data.

IoT node <NUM> may receive the data stream from IoT node <NUM> as data report <NUM>. Note that this may occur at any point including immediately after IoT node <NUM> receives data annotation configuration request <NUM>. For example, upon receipt of data annotation configuration request <NUM>, IoT node <NUM> may request to receive if necessary, and begin to buffer the data stream from IoT node <NUM> in preparation for annotating IoT node <NUM>'s data stream and transmitting IoT node <NUM>'s annotated data stream.

At block <NUM>, IoT node <NUM> annotates the data stream received from IoT node <NUM> based on the obtained and/or determined annotation concepts and/or data as determined at blocks <NUM>. IoT node <NUM> may subsequently transmit the annotated data stream to one or more IoT entities (not shown). At block <NUM>, IoT node <NUM> may store the annotated data stream and the determined annotation concept. IoT node <NUM> may also, or instead, forward the annotated data stream to other IoT nodes.

<FIG> shows exemplary, non-limiting signal flow <NUM> illustrating exemplary signals and procedures for requested data annotation (RDA). In some embodiments, an annotation requester, such as annotation requester <NUM>, which may be an IoT application, requests to add annotations to data stored at another node, such as IoT node <NUM> that may be an IoT server.

Annotation requester <NUM> may send data annotation request <NUM> to IoT node <NUM>. In some embodiments, data annotation request <NUM> may contain multiple annotation elements. Each annotation element may request annotations on different data (e.g., data items, data windows, data streams, partial data streams, etc.). Each annotation element may contain data such as a particular data stream(s) to be annotated, specific data item(s) to be annotated, and/or specific data windows to be annotated.

Data annotation request <NUM> may also, or instead, indicate a level of annotation to be used. For example, data annotation request <NUM> may include an instruction to use item-level annotation, window-level annotation, stream-level annotation, cross-partial-stream level annotation, and/or cross-stream allocation.

Data annotation request <NUM> may also, or instead, indicate an annotation concept or data to be used. For example, data annotation request <NUM> may explicitly indicate an annotation concept or data to be used in annotating the indicated data stream. Alternatively, or in addition, data annotation request <NUM> may include one or more links pointing to one or more annotation concepts or data that may be stored at an IoT concept server, such as IoT concept server <NUM>. Alternatively, or in addition, data annotation request <NUM> may indicate existing data streams from which annotation concepts or data may be determined and/or obtained. Alternatively, or in addition, data annotation request <NUM> may include a request for IoT node <NUM> to select and/or determine appropriate concepts and/or data for annotation of the indicated data stream(s).

At block <NUM>, IoT node <NUM> may analyze data annotation request <NUM>. As noted above, data annotation request <NUM> may include a link to one or more existing concepts or data stored at IoT concept server <NUM>. If so, IoT node <NUM> may send annotation concept request <NUM> to IoT concept server <NUM> requesting a reply that includes the specific concepts and/or data indicated in data annotation request <NUM>. In this embodiment, IoT concept server <NUM> may send annotation concept response <NUM> to IoT node <NUM> in response to receiving annotation concept request <NUM>.

At block <NUM>, IoT node <NUM> may use the annotation concepts and/or data obtained from data annotation request <NUM> and/or IoT concept server <NUM> to annotate the data items, data windows, data streams, and/or partial streams indicated in data annotation request <NUM>. Subsequently, IoT node <NUM> may send data annotation response <NUM> to annotation requestor <NUM> indicating whether the annotation was successful or not.

<FIG> shows exemplary, non-limiting signal flow <NUM> illustrating other exemplary signals and procedures for requested data annotation (RDA). In this embodiment, an IoT node, such as an IoT gateway, may request that another IoT node, such as an IoT server, add annotations to data stored on the requesting IoT node. For example, IoT node <NUM>, which may be an IoT gateway, may send data annotation request <NUM> to IoT node <NUM>, which may be an IoT server. Data annotation request <NUM> may include one or more annotation elements, each of which may request different annotations on different data. Each annotation element may include data indicating the data stream(s) on IoT node <NUM> to be annotated and abstract information about each of these data streams (e.g., description of the data stream), and the specific data items, windows, streams, and/or partial streams to be annotated.

At block <NUM>, IoT node <NUM> may analyze data annotation request <NUM> to determine an appropriate annotation concept and/or data, an annotation level to be used by IoT node <NUM>, and/or an annotation concept entity from which to obtain the annotation concept. IoT node <NUM>, in an embodiment upon determining an annotation concept entity based on request <NUM> or upon using some other means to determine an annotation concept entity, may send annotation concept request <NUM> to IoT semantic concept server <NUM> requesting concepts and/or data related to data annotation request <NUM>. IoT node <NUM> may receive annotation concepts and/or data from IoT semantic concept server <NUM> in annotation concept response <NUM>.

At block <NUM>, IoT node <NUM> may determine the data annotation results for the data streams indicated by data annotation request <NUM> based, at least in part in some embodiments, from data received from IoT semantic concept server <NUM> and/or data in data annotation request <NUM>. In some embodiments, IoT node <NUM> determines the annotation concept and/or data that should be applied to which particular data item, window, stream, etc. for each annotation element provided in data annotation request <NUM>.

Upon determination of the annotation concept, IoT node <NUM> may send data annotation response <NUM> to IoT node <NUM> that may contain multiple annotation result elements. Each annotation result element may contain the concept and/or data that should be added to the data item/window/stream associated with each annotation element specified in data annotation request <NUM>. In some embodiments, the annotation result element(s) within data annotation response <NUM> may explicitly indicate the annotation concept and/or data to be used. In other embodiments the annotation result element(s) within data annotation response <NUM> may provide a link pointing to one or more concept or data stored at an IoT semantic concept server, such as IoT semantic concept server <NUM>. In yet other embodiments, the annotation result element(s) within data annotation response <NUM> may indicate existing data streams from which annotation concepts may be determined and/or obtained. At block <NUM>, IoT node <NUM> applies annotations to the appropriate data items, data windows, and/or data streams. Note that in some embodiments, IoT node <NUM> may proactively send annotation results to IoT node <NUM>, in which case it may not be necessary to send data annotation request <NUM> and/or perform the processing of block <NUM>.

<FIG> shows exemplary, non-limiting signal flow <NUM> illustrating exemplary signals and procedures for annotation concept management. IoT node <NUM>, which may be an IoT device, gateway, or server, may send annotation concept access request <NUM> to IoT semantic concept server <NUM>. Annotation concept access request <NUM> may be used to perform various concept operations at IoT semantic concept server <NUM>, such as the discovery of one or more annotation concepts or data, the addition of one or more annotation concepts or data, the deletion of one or more annotation concepts or data, the retrieval of one or more annotation concepts or data, and/or the updating of one or more annotation concepts or data. Other additional operations may include combining multiple concepts and/or data and determining correlations among multiple concepts and/or data. IoT semantic concept server <NUM> may send annotation concept access response <NUM> to IoT node <NUM> indicating the success or failure of annotation concept access request <NUM>.

<FIG> shows DAaaS architecture <NUM> that may enable data annotation in M2M service platforms. DAaaS may be integrated into a device services entity (e.g., device services capability layer (DSCL) in ETSI M2M implementations or application service node common service entity (ASN-CSE) in oneM2M implementations), a gateway services entity (e.g., gateway services capability layer (GSCL) in ETSI M2M implementations or middle node common service entity (MN-CSE) in oneM2M implementations), and/or a network services entity (e.g., network services capability layer (NSCL) in ETSI M2M implementations or infrastructure node common service entity (IN-CSE) in oneM2M implementations) as a service capability or a service layer platform. In <FIG>, DAaaS <NUM> of device services entity <NUM>, which may be processing data stream <NUM>, may communicate with DAaaS <NUM> of gateway services entity <NUM>. DAaaS <NUM> of device services entity <NUM> may also communicate with DAaaS <NUM> of gateway services entity <NUM>. Device application (DA) <NUM> and/or network application (NA) <NUM> may communicate via interfaces with DAaaS <NUM> to initiate and/or manipulate the data annotation functionalities of DAaaS <NUM>. Note that device application (DA) <NUM> could be an application dedicated node application entity (ADN-AE) in oneM2M implementations. Note also that network application (NA) <NUM> could be an infrastructure node application entity (IN-AE) in oneM2M implementations. Note also that semantic concept server <NUM> may be integrated into network services entity <NUM>. Alternatively, semantic concept server <NUM> may be implemented as a standalone entity. A DAaaS in any services entity may be configured to communicate with semantic concept server <NUM>.

<FIG> illustrates exemplary non-limiting resource structure <NUM> that may be used to support DAaaS in some embodiments. For example, structure <NUM> may be used to implement the disclosed DAaaS embodiments within an ETSI M2M service architecture. Structure <NUM> may also be used in other architectures and implementations.

Base resource <NUM>, which, in an ETSI M2M implementation, may be an <sclBase> and/or <scl> resource that may be shared by other resources and in a oneM2M implementation may be an <CSEbase> and/or <remoteCSE> resource, may include collection resource concepts <NUM> that may include one or more concept resources that may be used to maintain one or more annotation concepts or annotation data, one of which is shown in <FIG> as sub-resource concept <NUM>. Concept <NUM> may be used to annotate or label a data item, data window, data stream, and/or multiple entire or partial data streams. Note that concept <NUM> may be associated with an event. Concept <NUM> may include attributes such as a description describing concept contained therein (e.g., "a person is preparing the lunch in the kitchen"), an expiration time indicating a time at which concept <NUM> will be no longer valid, and a link indicating that concept <NUM> is a virtual concept associated with a real concept that may be stored at the location indicated by the link. The value of the link attribute of concept <NUM> may be represented as a URI.

Structure <NUM> may also include containers resource <NUM>. Within containers <NUM> may be collection resource annotations <NUM> that may be used to maintain the annotations added to data items, data windows, data streams, and/or multiple entire or partial data streams. Note that in <FIG>, several annotations <NUM> resources are show to illustrate the various locations within structure <NUM> that annotations <NUM> may be configured. For example, annotations <NUM> may alternatively be a sub-resource of other resources, such as a content instances collection resource <NUM>, and/or a content instance resource. Annotations <NUM> may also be located directly under base resource <NUM> and may be shared by other resources, such as containers <NUM>, container <NUM>, content instances <NUM>, and/or content instance <NUM>.

Each annotation may be represented by sub-resource annotation <NUM> (shown to the right of annotations <NUM> in <FIG>). Each annotation resource indicates a notation label for the annotation and may include attributes such as annotation type that indicates the type of annotation (e.g., location, time, people activity, weather, appliance status, etc.), concept ID that identifies an existing concept resource, creation time that indicates when annotation <NUM> was created, expiration time that indicates the time at which annotation <NUM> will become invalid, and targeted data that indicated the resource on which annotation <NUM> is added. A targeted data attribute of annotation <NUM> may be optional when annotation <NUM> is configured under annotations <NUM> collection resource that is configured under containers collection resource <NUM> or under content instance resource <NUM>. Where annotation <NUM> is configured under annotations <NUM> collection resource that is configured under a container resource such as container <NUM>, the targeted data attribute may be used to indicate a data window to which the annotation resources is added.

Another resource that may be configured in structure <NUM> is collection resource windows <NUM> that may define data windows of a data stream. In an embodiment, a data stream may be represented by a container resource and therefore windows <NUM> may be a sub-resource of content instances <NUM> collection resource. Windows <NUM> may have one or more window <NUM> sub-resources, each of which may represent a data window of a data stream. A data window may contain one or more than one data item. Window <NUM> may have attributes such as a window size indicating the number of data items contained in the window represented by window <NUM>, a starting data item representing the first data item (that may be represented by a content instance resource) in the window represented by window <NUM>, and an ending data item that may represent the last data item (that may also be represented by a content instance resource) in the window represented by window <NUM>.

<FIG> is a diagram of an example M2M or IoT communication system <NUM> in which one or more disclosed embodiments of systems and methods for data annotation as a service may be implemented. Generally, M2M technologies provide building blocks for the IoT, and any M2M device, gateway, or service platform may be a component of the IoT as well as an IoT service layer, etc..

As shown in <FIG>, the M2M/IoT communication system <NUM> includes a communication network <NUM>. The communication network <NUM> may be a fixed network or a wireless network (e.g., WLAN, cellular, or the like) or a network of heterogeneous networks. For example, the communication network <NUM> may comprise of multiple access networks that provide content such as voice, data, video, messaging, broadcast, or the like to multiple users. For example, the communication network <NUM> may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), and the like. Further, the communication network <NUM> may comprise other networks such as a core network, the Internet, a sensor network, an industrial control network, a personal area network, a fused personal network, a satellite network, a home network, or an enterprise network for example.

As shown in <FIG>, the M2M/IoT communication system <NUM> may include an M2M gateway device <NUM>, and M2M terminal devices <NUM>. It will be appreciated that any number of M2M gateway devices <NUM> and M2M terminal devices <NUM> may be included in the M2M/IoT communication system <NUM> as desired. Each of the M2M gateway devices <NUM> and M2M terminal devices <NUM> may be configured to transmit and receive signals via the communication network <NUM> or direct radio link. The M2M gateway device <NUM> allows wireless M2M devices (e.g., cellular and non-cellular) as well as fixed network M2M devices (e.g., PLC) to communicate either through operator networks, such as the communication network <NUM>, or through direct radio link. For example, the M2M devices <NUM> may collect data and send the data, via the communication network <NUM> or direct radio link, to an M2M application <NUM> or M2M devices <NUM>. The M2M devices <NUM> may also receive data from the M2M application <NUM> or an M2M device <NUM>. Further, data and signals may be sent to and received from the M2M application <NUM> via an M2M service platform <NUM>, as described below. M2M devices <NUM> and gateways <NUM> may communicate via various networks including, cellular, WLAN, WPAN (e.g., Zigbee, 6LoWPAN, Bluetooth), direct radio link, and wireline for example. Any of the entities described herein, such as entities implementing DAaaS, annotation requesters, data stream generating entities, concept servers, IoT nodes, sensors, and any other entities and devices set forth herein, may be implemented, executed, or otherwise enabled, partially or entirely, on devices such as M2M devices <NUM>, gateways <NUM>, and service platform <NUM>. All such embodiments are contemplated as within the scope of the present disclosure.

The illustrated M2M service platform <NUM> provides services for the M2M application <NUM>, M2M gateway devices <NUM>, M2M terminal devices <NUM>, and the communication network <NUM>. It will be understood that the M2M service platform <NUM> may communicate with any number of M2M applications, M2M gateway devices <NUM>, M2M terminal devices <NUM>, and communication networks <NUM> as desired. The M2M service platform <NUM> may be implemented by one or more servers, computers, or the like. The M2M service platform <NUM> provides services such as management and monitoring of M2M terminal devices <NUM> and M2M gateway devices <NUM>. The M2M service platform <NUM> may also collect data and convert the data such that it is compatible with different types of M2M applications <NUM>. The functions of the M2M service platform <NUM> may be implemented in a variety of ways, for example as a web server, in the cellular core network, in the cloud, etc..

Referring also to <FIG>, the M2M service platform typically implements a service layer <NUM> (e.g. a network service capability layer (NSCL) as described herein) that provides a core set of service delivery capabilities that diverse applications and verticals can leverage. These service capabilities enable M2M applications <NUM> to interact with devices and perform functions such as data collection, data analysis, device management, security, billing, service/device discovery, etc. Essentially, these service capabilities free the applications of the burden of implementing these functionalities, thus simplifying application development and reducing cost and time to market. The service layer <NUM> also enables M2M applications <NUM> to communicate through various networks <NUM> in connection with the services that the service layer <NUM> provides.

In some embodiments, M2M applications <NUM> may include desired applications that form the basis for creation of one or more peer-to-peer networks that include devices that may use the disclosed of systems and methods for data annotation as a service. M2M applications <NUM> may include applications in various industries such as, without limitation, transportation, health and wellness, connected home, energy management, asset tracking, and security and surveillance. As mentioned above, the M2M service layer, running across the devices, gateways, and other servers of the system, supports functions such as, for example, data collection, device management, security, billing, location tracking/geofencing, device/service discovery, and legacy systems integration, and provides these functions as services to the M2M applications <NUM>. The applications with which the described service layer and objects interact may be applications such as those of M2M applications <NUM>.

<FIG> is a system diagram of an example M2M device <NUM>, such as an M2M terminal device <NUM> or an M2M gateway device <NUM> for example. As shown in <FIG>, the M2M device <NUM> may include a processor <NUM>, a transceiver <NUM>, a transmit/receive element <NUM>, a speaker/microphone <NUM>, a keypad <NUM>, a display/touchpad/indicators (e.g., one or more light emitting diodes (LEDs)) <NUM>, non-removable memory <NUM>, removable memory <NUM>, a power source <NUM>, a global positioning system (GPS) chipset <NUM>, and other peripherals <NUM>. It will be appreciated that the M2M device <NUM> may include any sub-combination of the foregoing elements while remaining consistent with an embodiment. This device may be a device that uses the disclosed systems and methods for data annotation as a service.

The processor <NUM> may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, one or more Application Specific Integrated Circuits (ASICs), one or more Field Programmable Gate Array (FPGAs) circuits, any other type and number of integrated circuits (ICs), a state machine, and the like. The processor <NUM> may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the M2M device <NUM> to operate in a wireless environment. The processor <NUM> may perform application-layer programs (e.g., browsers) and/or radio access-layer (RAN) programs and/or communications. The processor <NUM> may perform security operations such as authentication, security key agreement, and/or cryptographic operations, such as at the access layer and/or application layer for example.

The transmit/receive element <NUM> may be configured to transmit signals to, and/or receive signals from, an M2M service platform <NUM>. For example, in an embodiment, the transmit/receive element <NUM> may be an antenna configured to transmit and/or receive RF signals. The transmit/receive element <NUM> may support various networks and air interfaces, such as WLAN, WPAN, cellular, and the like. It will be appreciated that the transmit/receive element <NUM> may be configured to transmit and/or receive any combination of wireless or wired signals.

In addition, although the transmit/receive element <NUM> is depicted in <FIG> as a single element, the M2M device <NUM> may include any number of transmit/receive elements <NUM>. More specifically, the M2M device <NUM> may employ MIMO technology. Thus, in an embodiment, the M2M device <NUM> may include two or more transmit/receive elements <NUM> (e.g., multiple antennas) for transmitting and receiving wireless signals.

As noted above, the M2M device <NUM> may have multi-mode capabilities. Thus, the transceiver <NUM> may include multiple transceivers for enabling the M2M device <NUM> to communicate via multiple RATs, such as UTRA and IEEE <NUM>, for example.

The processor <NUM> may access information from, and store data in, any type of suitable memory, such as the non-removable memory <NUM> and/or the removable memory <NUM>. In other embodiments, the processor <NUM> may access information from, and store data in, memory that is not physically located on the M2M device <NUM>, such as on a server or a home computer. The processor <NUM> may be configured to control lighting patterns, images, or colors on the display or indicators <NUM> in response to various conditions and parameters, such as those described in some of embodiments set forth herein.

The processor <NUM> may receive power from the power source <NUM>, and may be configured to distribute and/or control the power to the other components in the M2M device <NUM>. The power source <NUM> may be any suitable device for powering the M2M device <NUM>.

The processor <NUM> may also be coupled to the GPS chipset <NUM>, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the M2M device <NUM>. It will be appreciated that the M2M device <NUM> may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

The processor <NUM> may further be coupled to other peripherals <NUM> that may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals <NUM> may include an accelerometer, an e-compass, a satellite transceiver, a sensor, a digital camera (for photographs or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like.

<FIG> is a block diagram of an exemplary computing system <NUM> on which, for example, the M2M service platform <NUM> of <FIG> and <FIG> may be implemented. Computing system <NUM> may comprise a computer or server and may be controlled primarily by computer readable instructions that may be in the form of software, wherever, or by whatever means such software is stored or accessed. Such computer readable instructions may be executed within central processing unit (CPU) <NUM> to cause computing system <NUM> to do work. In many known workstations, servers, and personal computers, central processing unit <NUM> is implemented by a single-chip CPU called a microprocessor. In other machines, the central processing unit <NUM> may comprise multiple processors. Coprocessor <NUM> is an optional processor, distinct from main CPU <NUM> that performs additional functions or assists CPU <NUM>. CPU <NUM> and/or coprocessor <NUM> may receive, generate, and process data related to the disclosed systems and methods for data annotation as a service.

In operation, CPU <NUM> fetches, decodes, and executes instructions, and transfers information to and from other resources via the computer's main data-transfer path, system bus <NUM>. Such a system bus connects the components in computing system <NUM> and defines the medium for data exchange. System bus <NUM> typically includes data lines for sending data, address lines for sending addresses, and control lines for sending interrupts and for operating the system bus. An example of such a system bus <NUM> is the PCI (Peripheral Component Interconnect) bus.

Memory devices coupled to system bus <NUM> include random access memory (RAM) <NUM> and read only memory (ROM) <NUM>. Such memories include circuitry that allows information to be stored and retrieved. ROMs <NUM> generally contain stored data that cannot easily be modified. Data stored in RAM <NUM> may be read or changed by CPU <NUM> or other hardware devices. Access to RAM <NUM> and/or ROM <NUM> may be controlled by memory controller <NUM>. Memory controller <NUM> may provide an address translation function that translates virtual addresses into physical addresses as instructions are executed. Memory controller <NUM> may also provide a memory protection function that isolates processes within the system and isolates system processes from user processes. Thus, a program running in a first mode can access only memory mapped by its own process virtual address space; it cannot access memory within another process's virtual address space unless memory sharing between the processes has been set up.

In addition, computing system <NUM> may contain peripherals controller <NUM> responsible for communicating instructions from CPU <NUM> to peripherals, such as printer <NUM>, keyboard <NUM>, mouse <NUM>, and disk drive <NUM>.

Display <NUM>, which is controlled by display controller <NUM>, is used to display visual output generated by computing system <NUM>. Such visual output may include text, graphics, animated graphics, and video. Display <NUM> may be implemented with a CRT-based video display, an LCD-based flat-panel display, gas plasma-based flat-panel display, or a touch-panel. Display controller <NUM> includes electronic components required to generate a video signal that is sent to display <NUM>.

Further, computing system <NUM> may contain network adaptor <NUM> that may be used to connect computing system <NUM> to an external communications network, such as network <NUM> of <FIG> and <FIG>. In an embodiment, network adaptor <NUM> may receive and transmit data related to the disclosed systems and methods for data annotation as a service.

It is understood that any or all of the systems, methods, and processes described herein may be embodied in the form of computer executable instructions (i.e., program code) stored on a computer-readable storage medium embodied as a physical device or apparatus. Such instructions, when executed by a machine, or a processor configured in a machine, such as a computer, server, M2M terminal device, M2M gateway device, or the like, effectuate, perform, and/or implement the systems, methods and processes described herein. Specifically, any of the steps, operations or functions described above may be implemented in the form of such computer executable instructions. Computer readable storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, but such computer readable storage media do not include signals. Computer readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other physical medium that can be used to store the desired information and that can be accessed by a computer.

In describing preferred embodiments of the subject matter of the present disclosure, as illustrated in the figures, specific terminology is employed for the sake of clarity.

Claim 1:
A computer-implemented method comprising:
receiving, at an Internet of Things, IoT, annotation entity (<NUM>) in a network of connected IoT entities (<NUM>), a first data stream from a first IoT entity (<NUM>) in the network of connected IoT entities (<NUM>);
determining (<NUM>) context information associated with the first data stream;
storing (<NUM>) the context information in an M2M service layer resource;
receiving, at the IoT annotation entity (<NUM>), a second data stream from a second IoT entity (<NUM>) in the network of connected IoT entities, the second data stream comprising multiple data items in multiple data windows;
determining (<NUM>), at the IoT annotation entity (<NUM>) and based on the context information, a data annotation to be applied to the second data stream at a window level; and
annotating (<NUM>) the second data stream at the IoT annotation entity (<NUM>) by applying the data annotation at the window level to one or more data windows, wherein at least one of the one or more data windows comprises a consecutive plurality of the data items defined by a size of that data window indicating the number of the plurality of the data items, a first data item of that data window, and an ending data item of that data window.