Patent Description:
Computer object detection encompasses a number of techniques in which sensor readings are used to identify physical objects in an environment. Sensors can include cameras (e.g., visual light cameras, infrared cameras, depth cameras, etc.), microphones, RADAR, LIDAR, ultrasonic sensors among others. Sensor readings alone generally do not provide object identification, but rather are processed to ascertain likely locations of an object and other characteristics that are used to identify the object.

Computer object detection is an underlying technology being used to enable autonomous vehicles or other assisted driving systems and vehicles. Generally, the computer object detection is combined to identify mobile and stationary objects in the environment of a vehicle, enable the vehicle to identify obstacles and navigate safely. Advances in autonomous and assisted vehicles have recently led to self-driving taxis or other conveyances that transport passengers without a human operator or attendant.

<CIT> relates to using ultrathin Bluetooth labels for asset tracking. <CIT> discloses that the dispenser of the wireless tape transmits a dispenser beacon to indicate the location of the dispenser and a geofence around that location, and when the smart device is in an out-of-base state, the smart device may monitor for beacons from one or more wireless tapes. <CIT> discloses that when the smart device receives the beacon from the wireless tape, the smart device may then store an indication in the local database records that the item tracked by the wireless tape is present.

Commuters lose belongings when using public transport. These belongings may include cellphones, wallets, keys, luggage, bags, clothing, or other items left in public transport such as taxis, robotaxis, buses, or trains. Causes for the loss may include forgetfulness, carelessness, accidentally dropping the items, or harsh traffic conditions (e.g., braking or swerving) that may cause the items to fall. In addition, autonomous delivery robots might lose cargo unnoticed.

Whenever a passenger loses their belongings, it is only natural that it causes concern to the individuals as their forgotten belongings in transport might be lost forever. The passenger could be losing expensive or sentimental items. Often, it may be too late by the time the passenger realizes that their belongings are gone in the transport.

A study in London has shown that <NUM>,<NUM> items were found on the Transport for the London network in the year <NUM>-<NUM>. Over <NUM> million people use trains, tubes, buses, trams, and more in London daily. However, the study illustrates that <NUM>% of these items are never claimed. The current London Public Transport process to reclaim lost items involves contacting passengers fill out a form and then forty or more staff members in the London Public Transport will try and match the completed form to items that are found and registered with their system. This is a lengthy process, and many items are never reclaimed. The majority of items are kept for three months, after which the items will then be donated, recycled, or disposed of.

A common technique to address lost items involves a passenger providing a mobile number with details of the items they have lost to the transportation company in the hope that, if the driver finds the item, the passenger could be contacted immediately. However, there is no guarantee that the belongings will ever be found and returned. It is possible that, even if the item was successfully located, not all drivers are willing to return them to the owner, or other passengers might have taken advantage of the situation and carried the item off as their own. Alternatively, it is also possible that a later passenger wishes to return the item but has no idea how to locate the owner.

Although object detection is becoming more prevalent in modern vehicles, there is currently no technique to match the detected object to its rightful owner. That is, a technique to establish unique object-person relationships is lacking. Although a car monitoring system may be able to detect and identify an object and its type, these systems do not associate the object with the passenger to which it belongs, making it difficult to return a lost item in a vehicle to its rightful owner.

To address this problem, other techniques have been employed. For example, an artificial intelligence-enabled system that uses technology (e.g., computer vision, machine learning, and natural language processing) to search a global database of images and descriptions to match the item found by airplane employees to match to a missing items report. This technique uses image recognition to identify unique details of the missing item, such as brand, material, color. Then, by recognizing similar words in the missing items report description, it can make a match quickly. However, this relies on accurate reporting by the passenger and any matching occurs well after the item is lost.

A mobile phone company has introduced a technique whereby sensors are used to locate the phone within a car. The technique uses sensors in the vehicle to triangulate the position of the missing phone, similar to how GPS does on a bigger scale. It also employs the phone's sensors to aid in the location detection, by tracking the movement of the phone if it moves around. Additionally, the system may use internal cameras to scan the car for unusual objects in the car as the first step to locating the lost phone. However, this technique relies on pre-knowledge of the phone and control of the vehicle, causing issues in public transportation.

A technique to identify and assess object value in one's own vehicle has been proposed. Specifically, valuable items left behind in a personally owned vehicle are detected and a warning is triggered if the item exceeds a certain monetary threshold. This application is mainly intended to prevent theft (e.g., of the item or the car) by notifying the user when an expensive item is left in their car. Here, a near-infrared (NIR) is used camera with a <NUM> degree field-of-view (FOV) lens that is mounted just above the rearview mirror, which provides a complete view of the front seats and an almost complete view of the rear seats. Items are detected by the camera and valued based on appearance. Training image generation techniques are used to produce a suitable synthetic dataset for object detection. Item price is used as an objective measure of value to help guide the detection process. Again, this technique is not conducive to public transportation environments because the relationship between the person and object is simply inferred by the person's ownership of the vehicle.

To address the issue of abandoned (e.g., left behind) items a technique to identify items and their owners in public spaces is described below. The technique operates by intelligently predicting and establishing a unique object-person relationship using computer vision and various sensors that are consolidated into a multimodal machine learning algorithm. The mechanism detects a disturbance in the object-person relationship and accurately predict whether passengers have left behind belongings in public transport, to proactively inform the passengers before even realizing the item was lost. The system may be applied to any mode of public transportation such as robotaxis, grab-cars, buses, trains, or even public spaces. In the case of delivery robots, the relationship comprises cargo items and the respective delivery robot losing these items, and detection of loss event could be realized by road side units (RSUs) or any other camera system e.g. traffic or location-based surveillance cameras. Additional examples and details are provided below.

A method according to the invention is defined in claim <NUM>, while a machine-readable medium comprising instructions to carry out the method according to the invention is defined in claim <NUM> and a corresponding apparatus is defined in claim <NUM>.

The dependent claims define further advantageous embodiments.

<FIG> is an example of an environment including a system for abandoned object detection, according to an embodiment. The illustrated system includes processing circuitry <NUM> and a sensor array <NUM> covering an area <NUM>. As illustrated, the person <NUM> was sitting in the public sofa <NUM> and placed her sunglasses <NUM> on the sofa <NUM>. She then proceeded to move <NUM> outside of the area <NUM> leaving her sunglasses <NUM> behind. This provides an example framework to illustrate the operation of the system.

The processing circuitry <NUM> is configured to establish a fence about (e.g., around) the person <NUM>. Here, "fence" is used to denote a boundary within which an item is designated as the person's and outside of which a detected object is not, akin to a geofence about a person. Thus, in general, fences are specific to people. Further, the fence moves with the person. Thus, in examples that follow, if an object is within a fence, the object may later be outside of the fence either because the object moved (e.g., and the person did not move), the person moved (e.g., and the object did not move), or both the person and the object moved. In an example, establishing the fence about the person includes registering the person at location, such as the public sofa <NUM>, a transportation terminal, or in a vehicle. In an example, the registration is performed at a vehicle at the beginning of a ride that the person is taking in the vehicle. Thus, when getting into a robotaxi, for example, the person <NUM> registers that they are entering the robotaxi. In an example, the registration is for a specific seat within the location.

In an example, establishing the fence includes dividing up a physical space based on the number of people occupying the physical space. This example addresses a situation where multiple people occupy a single area (e.g., the backseat of a taxi). Generally, if the person <NUM> were the only person in the backseat, the fence could be the entire seat, as it is common for people to spread belongings out in a given space. However, if another occupant enters the taxi, then the two people will likely restrict themselves to their own sides of the seat. Accordingly, the fences adjust to divide a space given the occupants of the physical space. In an example, the division is equal. In an example, the division accounts for the size of the occupants, providing an extent around each. Here, bigger people will have fences that cover a larger area.

In an example, the fence is terminated when the person <NUM> leaves the area <NUM>. Here, it is possible that the person <NUM> has forgotten her sunglasses when she leaves the area, providing a convenient time to be alerted to the mistake and retrieve the sunglasses <NUM>. In an example, the fence is terminated when the ride completes. This example contemplates alerting the person <NUM> proactively about the items that has been placed in the vehicle and has migrated beyond the fence. For example, a phone placed on the seat may have fallen to the floor during a braking or swerving event. As the ride completes, the person <NUM> is notified that the object is on the floor to retrieve before leaving the vehicle.

In an example, the fence is one of several fences. Here, each fence of the several fences may corresponding to a class of objects. Object classifications may be used to account for varying levels of distance from the person <NUM> and the object (e.g., sunglasses <NUM>) tolerated by people. For example, an animal (e.g., a pet) is often permitted its own space and may range farther than a phone. Conversely, an expensive watch may rarely be placed very far from the person <NUM> unless by mistake. The object classifications may address such factors as cost, convenience to replace (e.g., most food stuffs or snacks), and use patterns for the object to adjust how far away from the person <NUM> the fence extends.

The processing circuitry <NUM> is configured to detect the object (e.g., sunglasses) within sensor data provided by the sensor array <NUM> (e.g., received at an interface of the processing circuitry <NUM> to the sensor array <NUM>). The processing circuitry <NUM> tracks the distance between the person <NUM> and the object. In an example, this tracking is performed by detecting the object within the fence. When the object is within the fence when detected, it is ascribed to the person <NUM>.

In an example, the sensor array <NUM> includes a camera, RADAR, LIDAR, or ultrasonic sensors. In an example, detecting the object within the fence includes using the camera of the sensor array <NUM> to capture an image of the area <NUM>. The processing circuitry <NUM> is configured to detect and classify the object from the image. In an example, the processing circuitry <NUM> is configured to measuring a distance between the person and the object. This distance may be used solely as the basis for determining whether or not the object belongs to the person <NUM>. In this case, each object and person relationship may be considered a separate fence.

In an example, to detect that the object is outside of the fence, the processing circuitry <NUM> is configured to detect that an updated position between the person and the object exceeds a threshold based on the fence. Here, the fences may be the threshold, or the fence distance may be modified by an additional factor to arrive at the threshold. For example, the threshold may include a sensitivity factor that expands the fence. The sensitivity factor may address some things that are context dependent. Such as the person <NUM> putting a phone on the seat next to them and having it fall to the ground. If the object were trash, it would not be urgent to alert the person <NUM> during the ride. However, a phone, generally being more important to people, may warrant an alert when it first falls to the ground.

The processing circuitry <NUM> is configured to create an entry in an object-person relationship data structure to establish a relationship between the person <NUM> and the object within the fence. Here, the relationship is codified in the data structure establishing ownership of the object to the person <NUM> based on the initial proximity of the two when the object was detected. In an example, creating the entry in the object-person relationship data structure includes storing the distance between the person and the object. Generally, this distance is a physical distance (e.g., inches, centimeters, feet, etc.) but the distance may also be recorded in a consistent unit of measurement. Thus, for example, if the measurement is always performed from images produced by the same camera, the distance may be recorded in pixels from the image.

The processing circuitry <NUM> is configured to monitor positions of the object until an indication is received that the fence is terminated. Here, fence termination is an end condition at which point the monitoring ceases. Example of the end condition may be explicit (e.g., the person <NUM> ends a ride, the vehicle ends a ride, a stop is reached, etc.) or implicit (e.g., the person <NUM> leaves the area <NUM>).

In an example, monitoring the object includes updating the entry in the object-person relationship data structure with an updated distance between the person and the object. Here, the monitoring continues to detect the object and adjust the fence based on movement of the person <NUM>. The distance between the two is recorded, ultimately determining whether the object is within or without the fence. In an example, to perform the monitoring, the processing circuitry <NUM> is configured to repeatedly shut down the camera used to detect the object and employ at least one of an accelerometer, a motion sensor, or a pressure sensor in the area <NUM> to detect a change in state of an environment about the person <NUM>. The processing circuitry <NUM> may then activate the camera to detect the object again and measure a distance between the person <NUM> and the object. This example illustrates a way to save power in power constrained environments, such as in robotaxis. Here, the camera is used to actually detect the object and perform distance measurements to the person <NUM>. However, to save power, the camera is shut down. While the camera is not operating, lower power sensors are employed to detect state changes to the environment. For example, the accelerometer may detect a motion likely to provoke objects to slide or fall (e.g., a force in excess of 2Gs). A pressure sensor may be used to determine if the object has left a surface or been placed on a surface. Similarly, a motion sensor may simply note that things are moving within the area <NUM>. Once the lower power sensor provides evidence that objects may have moved, the processing circuitry <NUM> reactivates the camera to detect the objects again and update the distance relationship in the object-person relationship data structure.

When the processing circuitry <NUM> detects that the object is outside of the fence, the processing circuitry <NUM> is configured to alert the person <NUM> that the object is outside of the fence. In an example, the alert includes providing a message to a mobile phone of the person <NUM>. In an example, the alert includes actuating at least one of a visual, a haptic, or an auditory device proximate to the person. In these cases, a speaker, buzzer, or display is used to notify the user about the abandoned object. In an example, the actuator is placed within the area <NUM>, such as in the sofa <NUM>.

Using distance to establish relationships between the person <NUM> and objects is a flexible technique that conforms to usual behavior in public accommodation situations. Tracking this distance with respect to an acceptable extent (e.g., the fence) provides an efficient mechanism to determine when an object is likely to have been missed by the person <NUM>. Alerting the person <NUM> of this changed condition enables the person <NUM> to react within the moment to retrieve the object and avoid the losses that generally plague people using public accommodations, such as trains, airplanes, terminals, robotaxis, etc..

<FIG> is a block diagram of an example of a system for abandoned object detection, according to an embodiment. As illustrated, a variety of sensors-such as a vision sensor <NUM> (e.g., a camera), a motion sensor <NUM> (e.g., ultrasonic detector), an accelerometer or gyroscopic sensor <NUM>, a pressure sensor <NUM>-may be communicatively connected to processing circuitry <NUM> to provide data for analysis of the environment. The processing circuitry <NUM> may also be communicatively coupled to vehicle computers, such as the advanced driver-assistance system (ADAS) electronic control unit (ECU) <NUM> or connectivity ECU <NUM> to enable trip data or communication with a mobile phone of the person.

The processing circuitry <NUM> is configured to fuse the sensor data in a hardwired trigger <NUM> to determine when to activate the vision sensor <NUM> to perform object-person relationship classification using the object-person relationship classifier <NUM>. The classifier <NUM> provides the relationship details (e.g., which objects are related to which people and distances between them) to the fence trigger circuitry <NUM>, which applies the fence definitions to the relationships. When an object falls outside of the fence, the fence trigger circuitry <NUM> is configured to a speaker <NUM> or haptic feedback <NUM> to alert the person.

The vision sensors <NUM> contribute to image recognition analysis and may be used to identify the passenger, objects, and its surrounding. Vision sensors <NUM> may include visible light cameras, depth cameras (e.g., stereo cameras, time-of-flight depth cameras, etc.), infrared cameras, etc..

Motion sensors <NUM> may be used for object movement analysis, detecting when an object moves, such as objects falling onto the floor or objects moving away from the passenger. Examples of motion sensors <NUM> may include ultrasonic sensors, infrared sensors, radar, a multi-array microphone, etc. Motion sensors may be particularly useful when dealing with objects that fall into a blind spot of the passenger or vision sensors. For example, the object slipping in between the seats or falling underneath the seat.

Pressure sensors <NUM> may be used to analyze passenger body movement and detect when the passenger sits down, stands up, leans to a side, or has any sudden movements. An example of such sensors <NUM> may include seating posture monitoring sensors. Such sensing is useful when recognizing passenger movements throughout the journey using weight on seats. Whenever the passenger has any movements, there is a potential risk that some belongings slipping out of a pocket, falling from a lap, or otherwise moving unexpectedly.

The accelerometer or gyroscope <NUM> may be used to analyze vehicle movement and used to detect vehicle speed and motion. This may be useful when the vehicle to traversing dynamic conditions-such as moving through a storm, on bumpy roads, etc.-likely to cause objects or people within the vehicle to shift. For example, if the vehicle makes a sharp turn or sudden halt, it is likely that objects shift or fall in the vehicle.

The ADAS ECU <NUM> contributes to trip data analysis and may be used to detect start and stop times for a trip, provide positioning-such as global position system (GPS), Galileo, or global navigation satellite system (GLONASS)-to determine the location and travel route of a journey. The ADAS ECU <NUM> may also provide a variety of timestamps, such as when the vehicle is halted for a long time before resuming the journey. This data is useful when details for a passenger's journey help to establish a relationship between objects and people. For example, the passenger is going through a route that has a lot of construction work, certain timestamps-such as start times-and GPS may be used to estimate the high-risk areas.

Connectivity ECU's <NUM> may contribute to passenger data analysis and used to detect the passenger details during a registration process. This may be used to assign a passenger_ID or seat_ID. Such registration may also enable identification of the person for later contact in cases where an object is lost. In an example, it is useful to set a unique seating position to each passenger and acknowledge when each seat that is booked or assigned to them. With this data, the passenger_ID may be used as the base key for the object-person relationship data structure. Additionally, the Connectivity ECU <NUM> may identify which seat to perform sensory data analysis for a particular passenger or enable transmission of alerts to the passenger via the unique seat's actuators if necessary.

<FIG> illustrates an example of a sensor array in a vehicle, according to an embodiment. Here, multiple sensors with different modes of operation are strategically placed throughout the vehicle. The sensors include vision (e.g., camera <NUM>), motion (e.g., ultrasonic sensor <NUM>), pressure (e.g., pressure sensors 310A and 310B), and accelerometer + gyroscope (e.g., sensor <NUM> which may be included in an ECU of the vehicle).

<FIG> illustrates an example of a technique to establish object-person relationships, according to an embodiment. The technique may be considered via three sub-portions: object preparation, data registration, and object-person relationship building.

An Object preparation sub-system may include components <NUM>-<NUM>. Here, data from multiple sensor inputs <NUM>-such as motion, pressure, or accelerometer and gyroscope sensors-are sent to early multi-sensor model data trigger circuitry <NUM> (e.g., detection circuitry). Whenever there is a change in the sensor's data that exceeds a defined threshold value, detection level trigger is issued. In this case, an interesting or significant behavior in these sensors is detected and fed down the line for further analysis in the multi-sensor model data trigger circuitry <NUM> for object detection <NUM> using vision sensor data <NUM>. For example, a sudden spike in the accelerometer over a certain threshold may be flagged as high risk and be detected as a significant sensory input change.

The vision input frame <NUM> is used for object detection <NUM> and classification <NUM>, the outputs of which may be used for data fusion <NUM> and ultimately object localization within the environment. As noted elsewhere, the vision input frames may be sporadic to save power. Thus, a camera may periodically capture video frames with predefined interval time. Further, the camera may be triggered by the early detection circuitry <NUM> to performing intelligent enablement of the object detection circuitry <NUM>.

The data registration sub-system includes components <NUM> and <NUM>. These components ensure capture of information as vehicle trip details-such as Trip-Status, GPS, or vehicle_ID-and passenger details-such as passenger_ID. This data may be extracted from an ADAS ECUs or Connectivity ECUs of a vehicle. With the information obtained from both the passenger and vehicle ADAS, additional details about the passenger may be used to establish object-person relationships.

The object-person building subsystem includes components <NUM>-<NUM>. This subsystem uses the data from the object preparation and data registration sub-systems to build a detailed object-person relationship data structure. In operation, the subsystem labels object-IDs <NUM>. Based on the sensor data (e.g., object detection <NUM>, classification <NUM>, or localization <NUM>) a relationship is established between objects and a passenger. Each object is assigned a unique object_ID.

A distance is computed between the object and the person <NUM>. Thus, after collecting the detailed data from the ECU's and assigning each object with an object_ID, every object's distance to the passenger is calculated, for example, by using existing floor to plans of the vehicle.

Fence-off coordinates are computed <NUM> with details of the passenger_ID and each object's distance (e.g., (x,y,z) or (x,y)) data based on the passenger's geo-location and surrounding environment. This process constantly monitors the passenger and detects movement of the passenger changing fence boundaries around the passenger.

The data structure of the object-person relationship is built <NUM> that defines all of the passenger's unique details with the object associated with them. The data structure may also include the objects' type, category, or each object's (x,y) position in relation to the person.

The potential object-person relationship is then monitored <NUM> to detect potential breakage of the relationship between the objects and the passenger based on the configured distance value. Thus, if the object assumed to be the persons moves too far away from the person (e.g., outside of the fence), the person may be alerted.

In establishing object-person relationships, a distance model may be used. Consider a generic graph data structure comprising edges and vertices. Here, relationships may be modeled as edges with person_IDs and object_IDs as vertices. The following pseudocode demonstrates an algorithm to establish object-person relationship that fuses distance data, by using a graph and a kNN classifier for nearest neighbor object classification.

At the end of the for loop, a weighted graph is constructed with the following example code:
<IMG>.

The distance data above is an example of relationship data. Wearable objects-such as spectacles or watches-are generally are close to a person, whereas tools-such as cell phones-have a moderate distance to a person, and accessories-such as bags have a further distance to a person. Thus, the corresponding calculation may be referenced, as: <MAT>.

If R-value is more than <NUM>, the reference parameter is violated.

The below example shows how distance is used as R data to detect object misplace scenarios:.

<FIG> illustrates example interactions between components when creating object-person relationships, according to an embodiment. When the passenger is seated, passenger details may be encapsulated into a [Passenger_ID]. In an object detection phase, once the passenger is identified, object detection and classification are performed. The results are encapsulated into each object_ID for the respective objects detected.

At the object-person relationship phase, the object's distance with the person is processed to form the relationship. Each object's distance details are used to define the object-person relationship. This gets further encapsulated data into [Object_ID + Relationship].

Now, the object-person relationship data structure entry is created that compiles each data point to create the final data encapsulation of [Object_ID + Passenger_ID + Relationship].

<FIG> illustrates an example of on-boarding a passenger, according to an embodiment. The passenger on-boarding phase addresses what happens when the passenger first intends to take a vehicle or public transport to get to a destination. For example, a passenger wishes to travel from her house to the airport. She books a robotaxi with the help of the mobile app. When the vehicle arrives at the pick-up point (action <NUM>), the passenger opens the door and sits (action <NUM>). The passenger uses her mobile phone to register with the robotaxi through the connectivity ECU (action <NUM>). The car seat that the passenger has chosen to sit in then registers with the car's- connectivity ECUs (action <NUM>) once it detects that the passenger is seated (e.g., detected via pressure sensors). The trip-status and seat_ID, once assigned, may be retrieved for use later from the ECU.

<FIG> illustrates an example of distance measurements between a passenger and objects to establish object-person relationships, according to an embodiment. As noted above, this is used to build the object-person relationships. As illustrated, the person <NUM> has a distance relationship to several objects with unique IDs, including object <NUM> associated with object_ID <NUM>. The dashed line indicates the object <NUM> at a new position <NUM> at a later time (e.g., the object <NUM> is from an initial distance to a new distance from the person <NUM>).

Object detection begins when the person <NUM> sits on a particular seat. This may be the first data set used to build as the initial version of the data structure. In an example, using a machine learning training data set, a variety of objects may be classified, such as a handbag, luggage, phone, ring, watch, sunglasses, hat, or pets among others. As illustrated, each of these objects is assigned a unique Object_ID.

After object detection and classification is complete and a passenger-ID is assigned, object-person relationships may be established. The distance between a particular object to the person <NUM> may be determined using image analysis and geo-positioning to determine the location of the person <NUM> and object respectively. Next, (x,y) data that represents object distance from the person <NUM> may be incorporated into each object_ID with the person <NUM>. In an example, a map may be created with the different object_IDs and the person <NUM> including corresponding (x,y) distance data.

Monitoring is present throughout the person's journey. In an example, the motion and image sensors detect an object moving (e.g., object <NUM> moving to position <NUM>) such as sunglasses falling off of luggage. This updated distance will modify the entry for the object <NUM> in the data structure.

<FIG> illustrates entity relationships in a data structure for object-person relationships, according to an embodiment. Although the illustrated entity relationships are self-explaining, it is noted that the object-person relationship entity incorporates the passenger and objects (e.g., including object type or classification) and the distance between the objects and the person. As noted above, the initial data structure may be built upon the passenger's choice of a seat. Once an initial version of the data structure is created, updates to distances are recorded as well as possible status changes, such as the object falling outside of a pertinent fence.

<FIG> illustrate an example technique to , according to an embodiment. <FIG> walks through a typical use case. At first, a passenger begins the onboarding process and enters the vehicle (e.g., car, bus, train, etc.) and sits in a specific place (operation <NUM>). Object detection and classification (operation <NUM>) follows. Each object identified is associated with the person seated in each specific place. The training dataset is used to accurately perform object identification and classification.

Next, the initial version of the object-person relationship is built with the passenger model (operation <NUM>). All objects associated with the passenger are registered and stored in the object-person relationship data structure.

Once the initial version of the data structure is created, continuous monitoring is entered at operation <NUM>. That is, continuous monitoring begins with an idle (e.g., deactivated, in low-power mode, etc.) vision sensor.

Non-vision sensors-such as motion, accelerometer + gyroscope, or pressure sensors-actively monitor the environment (operations <NUM>, <NUM>, or <NUM>) for changes beyond a threshold. Once any of these sensors detect a change (decision <NUM>, <NUM>, or <NUM>), the vision sensor is activated (operation <NUM> to capture image data. In an example, the vision sensor is periodically activated capture an image frame (e.g., every N seconds) to monitor the objects if no activity on the non-vision sensors has been detected for some time.

The initial object-person relationship data structure is then retrieved (operation <NUM>). The new object-person relationship data is processed and compared to the initial version. If nothing has changed, the technique proceeds to the idle camera state <NUM>. Otherwise, an update is triggered to update the latest relationship data into the object-person data structure (operation <NUM>).

Monitoring of the status of the fence is also performed (decision <NUM>). As long as the fence is on (or in effect), the process continues at the idle camera operation <NUM>. When the passenger fence is off-e.g., as determined from a geofencing ratio given to the passenger with the aid of a vision sensor-the final object-person relationship version is extracted from the object-person relationship data structure along with the passenger details (operation <NUM>). Then, an alerting event (e.g., alert) may be triggered. In an example, the alert uses actuators-such as speaker, buzzers, display, or text-to-speech notifications-to notify the passenger of the missing item. In an example, the alert includes the passenger's name or a list of the missing items.

<FIG> illustrates a technique to notify a passenger of an abandoned object, according to an embodiment. This may also be called a passenger exit phase, such as when a passenger exits a vehicle or public area. In an example, geo-fencing mechanisms are used to detect when the passenger decides to leave the vehicle. As shown in <FIG> (area <NUM>), a threshold may be defined to indicate whether the passenger is leaving or not. Such movement (e.g., move <NUM> from <FIG>) may triggers the exit phase, resulting in retrieval of a final relationship status from the object-person relationship data structure.

The exits phase may include a configurable for sensitivity of each object (or type of object) in the object-person relationship (operation <NUM>). The sensitivity may inform a trigger threshold value for each assigned object. If a higher sensitivity threshold value is configured, then the object-person relationship becomes stricter. For example, a higher sensitivity value may be set for expensive belongings.

Continuous monitoring (decision <NUM>) is used to check on the object-person relationship. If there is no change, then the technique proceeds to decision <NUM>. Otherwise, the technique proceeds to decision <NUM>. At decision <NUM>, a process may wait on changes to be detected. In this case, the object-person relationship data structure provides detailed distance data to check on the sensitivity threshold for changes worth noting.

If the object-person relationship has changed, then followed by relationship sensitivity threshold is exceeded than the configured value, necessary measures are undertaken (operation <NUM>). If the relationship was not broken or exceeded the sensitivity threshold value, the monitor continues until the passenger turns to fence off (decision <NUM>).

If the object-person relationship breaks exceeding the threshold, the passenger may be notified (operation <NUM>) based on which object is broken via any of the vehicle's actuators. When the fence is off, monitoring ends.

<FIG> illustrates a flow diagram of an example of a method <NUM> for abandoned object detection, according to an embodiment. The method <NUM> is implemented in computer hardware, such as that described above or below (e.g., processing circuitry).

At operation <NUM>, a fence is established about (e.g., around) a person. In an example, establishing the fence about the person includes registering the person at a vehicle at the beginning of a ride that the person is taking in the vehicle. In an example, the fence is terminated when the ride completes. In an example, establishing the fence about the person includes dividing up a physical space based on the number of people occupying the physical space.

In an example, the fence is one of several fences, each fence corresponding to a class of objects, the object being in one class of the class of objects.

At operation <NUM>, an object within the fence is detected. In an example, detecting the object within the fence includes using a camera to detect and classify the object. In an example, detecting the object includes measuring a distance between the person and the object.

In an example, detecting that the object is outside of the fence includes detecting that an updated position between the person and the object exceeds a threshold based on the fence. In an example, the threshold includes a sensitivity factor that expands the fence.

At operation <NUM>, an entry in an object-person relationship data structure is created to establish a relationship between the person and the object within the fence. In an example, creating the entry in the object-person relationship data structure includes storing the distance between the person and the object.

At operation <NUM>, a position of the object is monitored until an indication is received that the fence is terminated. In an example, monitoring the object includes updating the entry in the object-person relationship data structure with an updated distance between the person and the object.

In an example, monitoring the position of the object includes repeatedly: shutting down the camera used to detect the object; using at least one of an accelerometer, a motion sensor, or a pressure sensor to detect a change in state of an environment about the person; and activating the camera to detect the object again and measure a distance between the person and the object.

At operation <NUM>, based on the monitoring of operation <NUM>, a detection is made that the object is outside of the fence.

At operation <NUM>, the person is alerted that the object is outside of the fence. In an example, alerting the person includes providing a message to a mobile phone of the person. In an example, alerting the person includes actuating at least one of a visual, a haptic, or an auditory device proximate to the person.

<FIG> illustrates a block diagram of an example machine <NUM> upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. Examples, as described herein, may include, or may operate by, logic or a number of components, or mechanisms in the machine <NUM>. Circuitry (e.g., processing circuitry) is a collection of circuits implemented in tangible entities of the machine <NUM> that include hardware (e.g., simple circuits, gates, logic, etc.). Circuitry membership may be flexible over time. Circuitries include members that may, alone or in combination, perform specified operations when operating. In an example, hardware of the circuitry may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuitry may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a machine readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuitry in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, in an example, the machine readable medium elements are part of the circuitry or are communicatively coupled to the other components of the circuitry when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuitry. For example, under operation, execution units may be used in a first circuit of a first circuitry at one point in time and reused by a second circuit in the first circuitry, or by a third circuit in a second circuitry at a different time. Additional examples of these components with respect to the machine <NUM> follow.

The machine (e.g., computer system) <NUM> may include a hardware processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory <NUM>, a static memory (e.g., memory or storage for firmware, microcode, a basic-input-output (BIOS), unified extensible firmware interface (UEFI), etc.) <NUM>, and mass storage <NUM> (e.g., hard drives, tape drives, flash storage, or other block devices) some or all of which may communicate with each other via an interlink (e.g., bus) <NUM>. The machine <NUM> may include an output controller <NUM>, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

Registers of the processor <NUM>, the main memory <NUM>, the static memory <NUM>, or the mass storage <NUM> may be, or include, a machine readable medium <NUM> on which is stored one or more sets of data structures or instructions <NUM> (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions <NUM> may also reside, completely or at least partially, within any of registers of the processor <NUM>, the main memory <NUM>, the static memory <NUM>, or the mass storage <NUM> during execution thereof by the machine <NUM>. In an example, one or any combination of the hardware processor <NUM>, the main memory <NUM>, the static memory <NUM>, or the mass storage <NUM> may constitute the machine readable media <NUM>.

The term "machine readable medium" may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine <NUM> and that cause the machine <NUM> to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine readable medium examples may include solid-state memories, optical media, magnetic media, and signals (e.g., radio frequency signals, other photon based signals, sound signals, etc.). In an example, a non-transitory machine readable medium comprises a machine readable medium with a plurality of particles having invariant (e.g., rest) mass, and thus are compositions of matter. Accordingly, non-transitory machine-readable media are machine readable media that do not include transitory propagating signals. Specific examples of non-transitory machine readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

In an example, information stored or otherwise provided on the machine readable medium <NUM> may be representative of the instructions <NUM>, such as instructions <NUM> themselves or a format from which the instructions <NUM> may be derived. This format from which the instructions <NUM> may be derived may include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., split into multiple packages), or the like. The information representative of the instructions <NUM> in the machine readable medium <NUM> may be processed by processing circuitry into the instructions to implement any of the operations discussed herein. For example, deriving the instructions <NUM> from the information (e.g., processing by the processing circuitry) may include: compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decoding, encrypting, unencrypting, packaging, unpackaging, or otherwise manipulating the information into the instructions <NUM>.

In an example, the derivation of the instructions <NUM> may include assembly, compilation, or interpretation of the information (e.g., by the processing circuitry) to create the instructions <NUM> from some intermediate or preprocessed format provided by the machine readable medium <NUM>. The information, when provided in multiple parts, may be combined, unpacked, and modified to create the instructions <NUM>. For example, the information may be in multiple compressed source code packages (or object code, or binary executable code, etc.) on one or several remote servers. The source code packages may be encrypted when in transit over a network and decrypted, uncompressed, assembled (e.g., linked) if necessary, and compiled or interpreted (e.g., into a library, stand-alone executable etc.) at a local machine, and executed by the local machine.

The instructions <NUM> may be further transmitted or received over a communications network <NUM> using a transmission medium via the network interface device <NUM> utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), LoRa/LoRaWAN, or satellite communication networks, mobile telephone networks (e.g., cellular networks such as those complying with <NUM>, <NUM> LTE/LTE-A, or <NUM> standards), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) <NUM> family of standards known as Wi-Fi®, IEEE <NUM> family of standards known as WiMax®, IEEE <NUM>. <NUM> family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device <NUM> may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network <NUM>. In an example, the network interface device <NUM> may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine <NUM>, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software. A transmission medium is a machine readable medium.

The drawings show, by way of illustration, specific embodiments that may be practiced. " Such examples may include elements in addition to those shown or described.

Claim 1:
A computer-implemented method for missing object detection, the method comprising:
establishing (<NUM>) a fence about a person;
detecting (<NUM>) an object within the fence;
creating (<NUM>) an entry in an object-person relationship data structure to establish a relationship between the person and the object within the fence;
monitoring (<NUM>) a position of the object until an indication is received that the fence is terminated;
detecting (<NUM>), from the monitoring, that the object has moved outside of the fence; and
alerting (<NUM>) the person that the object has moved outside of the fence.