Methods and apparatus for vehicle arrival notification based on object detection

A system is provided and generally includes a server and an associate computing device. The server may receive location video data comprising at least one image from a camera directed to a location, such as a designated area of a parking lot. The server may detect one or more objects in the image, and determine a bounding box based on the detected objects. The server may also determine a confidence value corresponding to the bounding box. The confidence value may be determined based on an area of the image defined by the bounding box and a predefined class. The server may then generate an arrival message based on the determined confidence value, and transmit the arrival message to the associate computing device.

TECHNICAL FIELD

The disclosure relates generally to customer service and, more specifically, to apparatus and methods for generating vehicle arrival notifications.

BACKGROUND

At least some known online retailers allow customers to place purchase orders remotely through a purchasing system. For example, some retailers use at least some known purchasing systems that allow customers to place purchase orders through the retailer's website. As another example, some retailers allow customers to place purchase orders through an application (“App”) that executes on a mobile device, such as a cellular phone. Often times, the purchasing system provides an option to the customer to have the purchased items delivered to an address, such as to the customer's home address. At least some known purchasing systems, however, provide an option to allow the customer to pick up the purchased goods at a store location. For example, the customer can select a store location when making the purchase, and have the retailer gather the goods and have them ready for customer pickup. Some purchasing systems allow the customer to check-in ahead of time to let the retailer know the customer is on their way to pick up the purchased goods at a store location.

In some examples, when the customer arrives at the store, the customer parks their vehicle and walks into the store. The customer then proceeds to customer service to pick up their purchased goods. In other examples, the customer parks their vehicle at a designated parking area and waits for an associate to assist the customer with their order. These customer service experiences, however, have drawbacks. For example, the amount of time before an associate is ready to assist a customer may be substantially long. The customer may also be unaware of how long they must wait before the associate is ready to help them with their order. As such, there are opportunities to address these and other customer service experiences.

SUMMARY

In some embodiments, a system is provided that includes a server, a video camera, and a computing device. The server is configured to receive video data comprising at least one image from an imaging device, such as a video camera. The server may detect at least one object in the at least one image. The server may then determine at least one area of the at least one image based on the detected object, where the determined area includes the detected object. The server may also determine a value, such as a confidence score, corresponding to the determined area of the image based on the detected object and a predefined class. The predefined class may be vehicles, for example. The server may then generate a vehicle arrival message based on the determined value. For example, the server may generate the vehicle arrival message if the value is beyond a predefined threshold. The server may then transmit the arrival message to the computing device, which may be operated by an associate of a retailer.

In some embodiments, a method by a server includes receiving video data comprising at least one image from an imaging device. The method further includes detecting at least one object in the at least one image. The method may also include determining at least one area of the at least one image based on the detected object, where the determined area includes the detected object. The method may further include determining a value, such as a confidence score, corresponding to the determined area of the image based on the detected object and a predefined class. The method may also include generating a vehicle arrival message based on the determined value. The method may further include transmitting the arrival message to the computing device.

In some embodiments, a non-transitory, computer-readable storage medium includes executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including receiving video data comprising at least one image from an imaging device. The operations further include detecting at least one object in the at least one image. The operations may also include determining at least one area of the at least one image based on the detected object, where the determined area includes the detected object. The operations may further include determining a value, such as a confidence score, corresponding to the determined area of the image based on the detected object and a predefined class. The operations may also include generating a vehicle arrival message based on the determined value. The operations may further include transmitting the arrival message to the computing device.

DETAILED DESCRIPTION

The description of the preferred embodiments is intended to be read in connection with the accompanying drawings, which are to be considered part of the entire written description of these disclosures. While the present disclosure is susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and will be described in detail herein. The objectives and advantages of the claimed subject matter will become more apparent from the following detailed description of these exemplary embodiments in connection with the accompanying drawings.

It should be understood, however, that the present disclosure is not intended to be limited to the particular forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives that fall within the spirit and scope of these exemplary embodiments. The terms “couple,” “coupled,” “operatively coupled,” “operatively connected,” and the like should be broadly understood to refer to connecting devices or components together either mechanically, electrically, wired, wirelessly, or otherwise, such that the connection allows the pertinent devices or components to operate (e.g., communicate) with each other as intended by virtue of that relationship.

The embodiments described herein may enable a retailer to be notified when a customer's vehicle arrives at a pickup location, such as at a store, to allow the retailer to service the customer as soon as possible. As such, the embodiments may allow the retailer to provide a more efficient means for a customer to pick up purchased goods from a store. For example, the systems and methods described herein may advantageously assist customers and store personnel in completing transactions for outside-store pickup transactions by decreasing customer wait times.

Turning to the drawings,FIG.1illustrates a block diagram of a vehicle notification system100that includes server102, multiple customer computing devices104,106,108, multiple associate computing devices110,112,114, and database116, each operably connected to network118. Server102, first customer computing device104, second customer computing device106, Nth customer computing device108, first associate computing device110, computing device112, and computing device114can each be any suitable computing device that includes any hardware or hardware and software combination for processing data. In addition, each can transmit data to, and receive data from, network118.

For example, each of server102, multiple customer computing devices104,106,108, and multiple associate computing devices110,112,114can be a computer, a workstation, a laptop, a mobile device such as a cellular phone, a web server, an application server, a cloud-based server, or any other suitable device. Each can include, for example, one or more processors, one or more field-programmable gate arrays (FPGAs), one or more application-specific integrated circuits (ASICs), one or more state machines, digital circuitry, or any other suitable circuitry.

AlthoughFIG.1illustrates three customer computing devices104,106,108, vehicle notification system100can include any number of customer computing devices104,106,108. Similarly, althoughFIG.1illustrates three associate computing devices110,112,114, vehicle notification system100can include any number of associate computing devices110,112,114.

Server102can be a server operated by a retailer. Associate computing devices110,112,114can be computing devices operated by associates (e.g., employees) of the retailer. Customer computing devices104,106,108can be computing devices operated by customers of a retailer.

Server102is operable to communicate with database116over network118. For example, server102can store data to, and read data from, database116. Database116can be a remote storage device, such as a cloud-based server, a memory device on another application server, a networked computer, or any other suitable remote storage. Although shown remote to server102, in some examples database116can be a local storage device, such as a hard drive, a non-volatile memory, or a USB stick. For example, database116can be a storage device local to server102.

Network(s)118can be a WiFi® network, a cellular network such as a 3GPP® network, a Bluetooth® network, a satellite network, a wireless local area network (LAN), a network utilizing radio-frequency (RF) communication protocols, a Near Field Communication (NFC) network, a wireless Metropolitan Area Network (MAN) connecting multiple wireless LANs, a wide area network (WAN), or any other suitable network. Network118can provide access to, for example, the Internet.

Server102can also communicate with first customer computing device104, second customer computing device106, and Nth customer computing device108over network118. Similarly, first customer computing device104, second customer computing device106, and Nth customer computing device108are operable to communicate with server102over network118. For example, server102can receive data (e.g., messages) from, and transmit data to, first customer computing device104, second customer computing device106, and Nth customer computing device108.

Server102is also operable to communicate with first associate computing device110, second associate computing device112, and Nth associate computing device114. For example, server102can receive data from, and transmit data to, first associate computing device110, second associate computing device112, and Nth associate computing device114.

Vehicle notification system100may allow for the notification of a customer's vehicle when the vehicle arrives at a predetermined location, such as a parking lot or an area outside a store designated for picking up purchased goods. As such, vehicle notification system100may improve the customer service experience by reducing the amount of time a customer would otherwise wait to be serviced. For example, vehicle notification system100may notify a retailer that a new vehicle has arrived at the predetermined location to allow the retailer to service the customer arriving in the vehicle, thereby reducing an amount of time the customer may otherwise have to wait for assistance from the retailer. an order for customer pickup for a customer when vehicle notification system100notifies the retailer that the customer has arrived.

For example, server102may be configured to receive video data including images taken from a video camera directed at a location, such as a portion of a retailer's parking lot. The location may be, for example, a designated area where customers park their vehicles to pick up previously ordered goods. Server102may employ a feature extractor, such as one based on neural networks or artificial intelligence (AI), to detect one or more objects in the received images. For example, server102may apply a feature extraction algorithm to the images to detect the objects (e.g., feature extraction), where the feature extraction algorithm may be one supported by the OpenCV library, a histogram of oriented gradients (HOG) feature extraction algorithm, or any other suitable feature extraction algorithm. Server102may determine a bounding box for any detected objects. For example, the bounding box may identify the location of each of the detected objects within the image. In some examples, the bounding box includes at least four coordinate points identifying an area within the image.

Server102may also employ a classifier, such as a random decision forest (RDF) classifier, to classify the detected objects. Server102may execute the classifier to classify the objects detected by the feature extractor and determine which of the detected objects are of a particular classification, such as vehicles. Server102may also determine a confidence value (e.g., confidence score) for the classification of each of the detected objects. For example, a high confidence value (e.g., 1, or 100%) may indicate a high likelihood that the classification of the object is correct (e.g., the object identified is a vehicle), where a low confidence value (e.g., 0, or 0%) indicates a low likelihood that the classification of the object is correct (e.g., the object identified is not a vehicle).

Based on the bounding boxes and confidence values for each of the objects identified as of a particular type (e.g., a vehicle) in an image, server102may filter out (e.g., remove from consideration) one or more of the objects. In some examples, if the confidence value for an object identified as a particular type is below a threshold (e.g., 60%), server102may filter that object out. In some examples, if the area of the image defined by a bounding box coincides with an area of the image defined by another bounding box, server102may filter out one of the objects. In some examples, server102filters out the object associated with a lower confidence value. In some examples, an object is filtered out if its corresponding confidence value is below a threshold and at least a portion of the image defined by its corresponding bounding box coincides with at least a portion of a bounding box corresponding to another object.

Server102may also determine whether an object, identified as of a particular type (e.g., a vehicle), was identified as an object in a previous image. Server102may perform such operations to determine whether, for example, the object (e.g., the vehicle) has moved. If server102determines the object has moved, server102may filter out the object (e.g., as identified by the more current image). The determination may be based on one or more of an amount of time that has elapsed between when the previous and current images of the objects were taken or received, and the bounding boxes associated with the previous and current identified objects.

For example, if the previous image corresponding to the previous object identified was taken within a threshold amount of time (e.g., 5 minutes) from when the current image corresponding to the current object identified was taken, and the bounding boxes corresponding to the two images are within a threshold distance from each other, server102may determine that the identified objects are the same and may filter out one of the objects (e.g., the one corresponding to the newer image). Server102may determine whether the bounding boxes are within a threshold distance from each other by, for example, determining the distance between the center points of each of the bounding boxes.

In some examples, server102may determine whether the bounding boxes are within a threshold distance from each other based on the differences of corresponding coordinate points for the bounding boxes. For example, server102may determine the difference in distance in the vertical direction (e.g., y coordinate) between the top-left coordinate points of the bounding boxes, and the difference in distance in the horizontal direction (e.g., y coordinate) between the top-left coordinates. Server102may determine that the bounding boxes are within a threshold distance from each other if the vertical distance is within a vertical threshold, and the horizontal distance is within a horizontal threshold. Server102may determine similar distance differences for the top-right, bottom-left, and bottom-right coordinates of the bounding boxes. In some examples, the server102may determine that the bounding boxes are within a threshold distance from each other if all vertical and horizontal differences for each of the coordinate points of the bounding box are within vertical and horizontal thresholds, respectively.

If an object was identified as a particular type and was not filtered out, server102may transmit a message (e.g., vehicle arrival message) to another computing device, such as one or more of associate computing device110,112,144. The message may notify, for example, an associate of a retailer that a new vehicle has arrived at the location for which images where provided, such as a portion of a retailer's parking lot.

In some examples, in response to receiving the message, associate computing device110,112,114may present a notification to the associate (e.g., via a display unit) that a new customer has arrived at the location. The associate may then be aware that a new customer has arrived, and may proceed to the location to service the customer. In some examples, the associate computing device110,112,114may transmit a signal to server102when the associate is ready to service the customer. In response, server102may provide a signal to a display device, such as one visible by customers near the location, to cause the display device to display a message notifying customers that an associate is on their way (e.g., “Associate coming to assist you with your order.”).

FIG.2illustrates a more detailed view of the server102ofFIG.1. Server102can include one or more processors201, working memory202, one or more input/output devices203, instruction memory207, transceiver204, and a display206, all operatively coupled to one or more data buses208. Data buses208allow for communication among the various devices. Data buses208can include wired, or wireless, communication channels.

Processor(s)201are configured to perform a certain function or operation by executing code, stored on instruction memory207, embodying the function or operation. For example, processor(s)201can be configured to perform one or more of any function, method, or operation disclosed herein.

Instruction memory207can store instructions that can be accessed (e.g., read) and executed by processor(s)201. For example, instruction memory207can include read-only memory (ROM) such as electrically erasable programmable read-only memory (EEPROM), flash memory, a removable disk, CD-ROM, any non-volatile memory, or any other suitable memory.

Processor(s)201can store data to, and read data from, working memory202. For example, processor(s)201can store a working set of instructions to working memory202, such as instructions loaded from instruction memory207. Processor(s)201can also use working memory202to store dynamic data created during the operation of server102. Working memory202can be a random access memory (RAM) such as a static random access memory (SRAM) or dynamic random access memory (DRAM), or any other suitable memory.

Input-output devices203can include any suitable device that allows for data input or output. For example, input-output devices203can include one or more of a keyboard, a touchpad, a mouse, a stylus, a touchscreen, a physical button, a speaker, a microphone, or any other suitable input or output device.

Display206can display user interface205. User interface205can enable user interaction with server102. In some examples, a user can interact with user interface205by engaging input-output devices203. Display206can be any suitable display. For example, display206can be a computer monitor. In some examples, display206includes a touchscreen.

Transceiver204allows for communication with a network, such as a network of network(s)118ofFIG.1. In some examples, transceiver204is selected based on the type of network(s)118server102will be operating in. For example, transceiver204can allow for communications with a WiFi® network in the example when network118is a WiFi® network. Processor(s)201is operable to receive data from, or send data to, network118via transceiver204.

FIG.3illustrates a block diagram illustrating various portions of the vehicle notification system100ofFIG.1. In this example, video camera304, which may be a security camera or any other suitable camera, is directed to location302. Location302may be, for example, a pre-defined location, such as a portion of a retailer's parking lot where customers park their vehicles to await delivery of previously ordered items. For example, the items may have been ordered on the retailer's website, and the customer may have selected to pick the items up at the retailer's location. Video camera304provides a video stream318of location302to server102.

As indicated in the figure, server102includes vehicle recognition engine306, vehicle detection engine308, vehicle determination engine310, and new vehicle identification engine312. In some examples, some or all of vehicle recognition engine306, vehicle detection engine308, vehicle determination engine310, and new vehicle identification engine312are implemented in processor(s)201ofFIG.2, and in executable instructions executed by processor(s)201. In some examples, some or all of vehicle recognition engine306, vehicle detection engine308, vehicle determination engine310, and new vehicle identification engine312can be implemented in hardware, such as digital circuitry, FPGAs, ASICs, state machines, or any other suitable hardware.

Vehicle recognition engine306is operable to receive video stream318from video camera304and extract images from the video stream318. Vehicle recognition engine306may store video image data314identifying the extracted video images in database116. Vehicle recognition engine306may determine one or more features of objects, such as vehicles, from the extracted images. For example, vehicle recognition engine306may employ a feature extractor, such as one based on neural networks or artificial intelligence (AI), to detect one or more objects in the extracted images. Vehicle recognition engine306may store vehicle feature data316identifying the extracted features in database116. For example, vehicle recognition engine306may store in database116vehicle feature data316that includes feature data for a first vehicle comprising a vehicle identification (ID) “V1” as well as an array of features, illustrated by “F1,” “F2,” and “F3.” The array of features for a vehicle can include any number of features, and can be features extracted from one or more images of the received video stream318. Vehicle recognition engine306may provide feature data320, which identifies one or more detected features in the extracted images, to vehicle detection engine308.

Vehicle detection engine308is operable to classify objects based on the feature data320received from vehicle recognition engine306. For example, vehicle detection engine308may include a classifier, such as an RDF classifier, to classify the objects identified by feature data320. Vehicle detection engine308may first be trained as a classifier to differentiate between vehicle and non-vehicle regions of a video image, for example. Vehicle detection engine308may determine which objects have been classified as of a particular type, e.g., vehicles, and may further identify a bounding box for each detected object. The bounding box may identify the location of each of the detected objects within the image. In some examples, the bounding box includes at least four coordinate points identifying an area within the received image containing an object. Vehicle detection engine308may also determine a confidence value for the classification of each of the detected objects. For example, vehicle detection engine308may determine a confidence value for a detected object based on how likely the detected object is a vehicle. Vehicle detection engine308may provide confidence score data324identifying confidence values for detected objects of a particular type (e.g., vehicles), and bounding box coordinate data326identifying a bounding box for each detected object of the particular type, to vehicle determination engine310.

Vehicle determination engine310may be operable to receive confidence score data324and bounding box coordinate data326corresponding to detected objects, and may filter out (e.g., remove from consideration) one or more of the objects. For example, vehicle determination engine310may determine whether a confidence value, identified by confidence score data324and indicating how likely a detected object within a corresponding bounding box identified by bounding box coordinate data326is a vehicle, is below a threshold (e.g., 60%). If the confidence value is below the threshold, vehicle determination engine310may remove that object from consideration. In some examples, if the area of the image defined by a bounding box coincides with an area of the image defined by another bounding box, vehicle determination engine310may filter out one of the objects. In some examples, vehicle determination engine310filters out the object associated with a lower confidence value. In some examples, vehicle determination engine310filters out an object if its corresponding confidence value is below a threshold and at least a portion of the image defined by its corresponding bounding box coincides with at least a portion of a bounding box corresponding to another object. Vehicle detection engine308may then provide filtered bounding box coordinate data311, which identifies bounding boxes for remaining objects (e.g., objects not filtered out by vehicle determination engine310), to new vehicle identification engine312.

New vehicle identification engine312may determine whether an object, identified as of a particular type (e.g., a vehicle), was identified as an object in a previous image. New vehicle identification engine312may determine whether, for example, the object (e.g., the vehicle) has moved. If new vehicle identification engine312determines the object has moved, new vehicle identification engine312may filter out the object (e.g., as identified by the more current image). New vehicle identification engine312may make the determination based on one or more of an amount of time that has elapsed between when the previous and current images of the objects were taken or received, and the bounding boxes, as identified by filtered bounding box coordinate data311, associated with the previous and current identified objects. As discussed further below, database116stores vehicle data314, which may include filtered bounding box coordinate data311and video image time data319associated with previously identified vehicles.

In some examples, new vehicle identification engine312determines the amount of time that has elapsed between when the previous and current images of the objects were taken based on the video image time data319, which identifies a time (e.g., timestamp) each image was taken. For example, vehicle recognition engine306may determine the time of each video image based on data, such as meta data, contained within video stream318. In some examples, new vehicle identification engine312determines whether a previous image, corresponding to a previously identified object, was taken within a threshold amount of time (e.g., 5 minutes) from when a current image, corresponding to a currently identified object, was taken.

New vehicle identification engine312may also determine whether the bounding boxes corresponding to the two images are within a threshold distance from each other. New vehicle identification engine312may determine whether the bounding boxes are within a threshold distance from each other by, for example, determining the distance between the center points of each of the bounding boxes. New vehicle identification engine312may determine that the identified objects are the same if the previous and current images were taken within the threshold amount of time, and the bounding boxes are within a threshold distance from each other. Server102may determine whether the bounding boxes are within a threshold distance from each other by, for example, determining the distance between the center points of each of the bounding boxes.

If new vehicle identification engine312determines that identified objects are the same, new vehicle identification engine312may filter out one of the objects (e.g., the one corresponding to the newer image). Otherwise, new vehicle identification engine312determines that the vehicle is a new arriving vehicle, and stores vehicle image data314identifying the vehicle in database116. For example, vehicle image data114may include the filtered bounding box coordinate data311for the new vehicle object. Vehicle image data114may also include, for example, the video image time data319corresponding to the new vehicle object.

In some examples, if new vehicle identification engine312determines that a detected vehicle is a new arriving vehicle, new vehicle identification engine312may transmit a vehicle arrival message320to one or more of associate computing device110,112,114. As such, a user of associate computing device110,112,114may be notified that a new vehicle has arrived to location302. For example, associate computing devices110,112,114may alert an associate that the customer has arrived to location302. For example, associate computing devices110,112,114may display a message alerting the user that the customer has arrived. In some examples, associate computing devices110,112,114vibrate or provide any other suitable notice to the user.

FIG.4illustrates another block diagram illustrating various portions of the vehicle notification system100ofFIG.1. In this example, arriving vehicles410arrive at location302, which is monitored by video camera304. Video camera304provides video stream318to server102, which includes one or more video images of location302. Server102may continuously monitor video stream318to detect one or more vehicles, such as arriving vehicles410. Server102may determine whether any of the arriving vehicles410are new arriving vehicles (e.g., they have not been reported previously). If server102determines an arriving vehicle is a new vehicle, server102may transmit a vehicle arrival message320to associate computing device110,112,114to notify one or more associates that a new vehicle has arrive and is awaiting service. The vehicle arrival message320may be, for example, an electronic email (email), a text message (e.g., short message service (SMS) message), or may be a mobile push of data to the associate computing device110,112,114.

In some examples, in response to the vehicle arrival message320, an associate identifies an order for a customer in arriving vehicles410, and provides input to one or more of associate computing device110,112,114to retrieve the order from server102. For example, the associate may ask the customer for the order number, and the associate may provide the order number to associate computing device110,112,114via, for example, an I/O device. Associate computing device110,112,114may then, in response to receiving the input, transmit a customer order request message416to server102. In response, server102may obtain customer order data312from database402, which identifies one or more orders for the customer. The associate may then provide the order contents to the customer.

As such, because associates408are notified once arriving vehicles410arrive to location302by vehicle notification system100, the amount of time customers in arriving vehicles410might otherwise wait to have their ordered items delivered to them is reduced.

FIG.5is a flowchart of an example method that can be carried out by a server, such as server102ofFIG.1. Beginning at step502, location video data is received. The location video data comprises at least one image from a camera directed to a pre-defined location. For example, server102may receive location video data from video camera304, which may be directed at location302. At step504, at least one object within an image is detected based on the location video data. For example, server102may detect one or more objects, such as vehicles, in video stream318.

Proceeding to step506, an area of the image is determined based on the detected object. For example, server102may determine a bounding box corresponding to the detected object. At step508, a value is determined based on the detected object and a predefined class, such as vehicles. For example, server102may determine a confidence value for the detected object indicating how likely the object detected is of a particular type, such as a vehicle.

At step510, an arrival message is generated based on the value. For example, server102may generate a vehicle arrival message320to be transmitted to one or more of associate computing devices110,112,114if the determined confidence value is beyond (e.g., over) a threshold. At step512, the arrival message is transmitted. For example, server102may transmit the vehicle arrival message320to one or more of associate computing devices110,112,114.

FIG.6is a flowchart of another example method that can be carried out by a server, such as server102ofFIG.1. Begging at step602, location video data is received. The location video data comprises at least one image from a video camera directed to a pre-defined location. For example, server102may receive location video data from video camera304, which may be directed at location302. At step604, a video image is extracted from the location video data. At step606, an object is detected in the video image. For example, server102may apply a feature extraction algorithm to the location video data to detect the object. At step608, a bounding box for the detected object is determined. For example, the bounding box may include coordinate points that identify an area of the image that includes the detected object. At step610, a confidence value is determined based on a predetermined class. The confidence value may be based on how likely the detected object within the bouncing box is a vehicle, for example.

At step612, a determination is made as to whether the detected object should be filtered out based on the bounding box and the confidence value. For example, server102may filter out the detected object if the confidence value is below a threshold and at least a portion of the image defined by the bounding box coincides with at least a portion of a bounding box corresponding to another object. If the detected object is filtered out, the method ends. Otherwise, the method proceeds to step614.

At step614, a determination is made as to whether the object detected has been previously identified. For example, server102may determine whether the detected object is a same vehicle that was previously reported within a previous amount of time (e.g., 5 minutes). If the detected object has been previously identified, the method ends. Otherwise, the method proceeds to step616.

At step616, an arrival message is generated and transmitted to another computing device. For example, server102may generate a vehicle arrival message320, and transmit it to one or more of associate computing devices110,112,114. The method then ends.