System and method for interactively reporting of roadway incidents on an AI device

System and methods for including a system mounted to a vehicle for identifying incidents of a roadway and transmitting the incidents to a server, the server located remotely from the system, the system comprising: a device having: a camera for obtaining digital images; at least one sensor including a location based sensor; a memory and processor for executing image processing instructions for processing the digital images for automated detection of the incidents, generating object data based on the processing, generating incident data including the object data and the images; and a network interface for sending the incident data over a communications network to the server during operation of the vehicle on the roadway. Also included are both interactive and autonomous possesses for implementing the device.

FIELD

The present invention is related to image acquisition for road related incidents.

BACKGROUND

Although many technologies exist for surveying the roads, oftentimes the costs and constraints make it less appealing for practical use. Many solutions require modifications to the vehicle, costly attachments, computers and training to use. Usually these technologies specialize in surveying a single component of road condition. To detect the full spectrum of incidents expected to keep roads maintained, the most common and effective practice is still largely manual. Surveyors drive vehicles down roads and stop their vehicle when they find an object of interest. They take a picture, and then note down the incident details to revisit. This is a slow and costly method, oftentimes leaving a backlog of roads that are overdue for reassessment. As municipalities grow, it becomes harder and harder to stay on top of all the incidents in the system.

Municipalities occasionally outsource road assessment to companies which utilize specialized vehicles employing a wide array of sensors, including cameras, thermal, vibration, radar and laser. While providing a greater level of detail, a specialized vehicle is required to complete the task. The vehicle sensors require extensive calibration, and the processing of the data requires specialized systems and knowledge to produce a report. The costs are extensive, and therefore, such assessments are typically done periodically, where the period duration typically ranges from one year to ten years.

While there have been occasional instances of applications aimed at identifying road based incidents in video or images, such processes are typically done by first acquiring a data set (in the form of video and geo-positioning data), and then uploading the data to a specialized server where it is processed. The process is cumbersome as often the dataset is too large to transmit over cellular networks. Thus, disadvantageously, current systems must rely upon access to the complete image dataset only once the vehicle has returned to the image processing facility.

Further, today, many fleet vehicles utilize dashcams. Dashcams are camera devices which are mounted on the windshield on a vehicle. Dashcams are typically used in order to capture video clips for situations where claims may take place. Dashcams can record video in vehicle or out of vehicle. Some example of use of dashcams in vehicles include identifying fault in collisions for insurance purposes and ensuring conformity and compliance to processes and procedures. Dashcams are typically used in any vehicle and are used in private vehicles, waste management vehicles, taxis and ridesharing vehicles, public service vehicles, snowplows, amongst others.

The data from the dashcams is stored locally on the device on a non-volatile memory, such as solid-state memory, disks, flash drives, hard drives, etc. The dashcam data is accessible through wired or wireless connection to a dash camera. It may be removed from a vehicle upon incident (such as the case when there is a collision), and connected to a PC using a USB cable. The video can also be wirelessly uploaded to a server at the end of a shift (as may be the case with vehicle fleets, such as snowplows and taxis).

However, it is recognised that uploading a large amount of data is not always the efficient way to access the data, as not all recorded data can be equally important. Dashcams can also be connected to a variety of a sensors which generate incidents in order to tag, or bookmark, chunks of the video which may be of interest. Examples may include duress buttons for taxi drivers, accelerometer events (i.e. rapid acceleration/deceleration, or collisions), amongst other types of sensors and incidents. Those events can then be automatically uploaded at the end of a trip using wifi, or be uploaded using a cellular connection.

Dashcams can also have driver facing cameras which monitor the driver's attention by using artificial intelligence (AI). They may note if the driver is holding something (food or beverage), note if the driver is falling asleep, holding a cellphone, or looking away from the windshield.

Artificially intelligent devices are nowadays making their way into vehicles for automated incident detection of objects of interest outside of a vehicle, and reporting incidents which require the attention of the responsible party to repair. Examples of deficiencies of interest including road cracking, deformations and/or distortions in road, road patches/seals, road damage, street signage issues, manhole issues, drainage issues, pavement marking issues, road obstruction issues, or sidewalk issues.

Such artificially intelligent devices can be called smart cameras. Smart cameras identify issues on a roadway at a rate and efficiency beyond those of human capabilities, and report them through video clips or images tagged with additional sensor data such as GPS coordinates. Smart cameras can note and map hundreds of potholes in the span of a short drive. However, smart cameras may also make mistakes and miss deficiencies which they have not seen before, or not properly trained to identify.

Training an AI model constitutes a large portion of the development time and determines what objects a model can detect, classify or segment. It is important to note that a model can only find objects that the model had been trained to find. Some incidents may not be that common, and as such it can be difficult to obtain a sufficient data sample in order to reliably train a model. Due to the vast amount of data required, and the time and effort to annotate datasets to train comprehensive models, having a camera being able to automatically identify every possible deficiency is an unrealistic proposition. Further, in some circumstances, the operator of an AI enabled device may have a lack of confidence in the inferred results of the device, in particular for those devices providing a continuous stream of real-time inferences and associated information.

SUMMARY

It is an object of the present invention to provide an image acquisition system and/or method to obviate or mitigate at least one of the above presented disadvantages.

Provided is a system and method that supplements or otherwise adjusts the use of a device with artificial intelligence capabilities, interactively, together with incidents logged by a human operator of the device, for providing a system that simultaneously can identify deficiencies and issues of interest on a roadway using artificial intelligence and human input/interaction. The system and method are deployed to a vehicle, where the system automates collection of road related objects and/or incidents. A user of such a system can encounter scenarios where they would want to capture data (image, GPS/GNSS position etc.) of an object or incident, but the artificial neural networks of the system may have not been trained for the particular object or incident that the operators are interested in, or do not provide this functionality at an acceptable level of accuracy. It also may be an advantage for the operator to manually adjust/interact with a previously captured image, or otherwise manually record and adjust a newly captured image, in order to facilitate a more complete assessment of the roadway and surrounding conditions.

In order to interactively tag relevant data of such potentially untrained incidents using the AI device, further functionality is provided by artificially intelligent systems to supplement operator intervention (e.g. adjustment/interaction) by tagging of event(s) preferably without interfering with the core AI functions as programmed. In this manner, it is advantageous to have a plurality of different processes performed by the AI device, e.g. manually interactive as well as autonomous, such that the different processes can be done in parallel and/or in series. It is also recognised that the operator can choose to skip or otherwise bypass for one or more selected images the autonomous AI image processing, or any parts thereof.

Current technologies for surveying roads can be extremely costly, requiring complex installation of specialized sensors on a vehicle. These technologies require extensive training and experience to operate and are typically limited only to analysis of pavement condition. Due to the high cost and limited capabilities of the available technologies, municipalities are currently relying on simple manual inspection to cover the detection of all road related incidents. Further, existing on board imaging systems used for navigation of a vehicle are not optimized for acquiring appropriate images containing objects of interest with respect to a road surface. As such, using a specialized device containing a camera that is mounted to the vehicle is desired.

Municipalities send their maintenance workers to drive the service vehicles on roads where they must visually locate road related concerns while on patrol or a service call. When workers spot a potential concern they are required to pull over and get off the vehicle to closely investigate the issue and manually initiate an incident or a work order using whatever system they have in place. It may be logged by pen and paper or utilizing a computerized tablet or smart phone.

Getting off the vehicle for each detected issue is not only time consuming but also put workers in a dangerous situation. Contrary to the current technologies available and the method that is implemented to detect road related incidents, the system provided here is capable of automated identification and reporting of a wide variety of road related incidents (e.g. various types of pavement damage and degradation, road sign damage, road debris, obstructions and other incidents visible from inside of a vehicle), including operator implemented adjustment while the proposed system is in operation. It is also disadvantageous to require the operator to manually record and comment on every picture taken. Also, it is disadvantageous for the operator to have no ability to interact with capturing/processing of images in AI enabled autonomous processing of images during navigation of roadways and surroundings.

The system can comprise: device—a mobile computing device such as smartphone, or embedded computer system with built in or peripheral camera(s) and application utilizing neural network(s) and machine learning work together with server(s) that process and store data received from single or multiple devices, serve as gateway to users via web access and present data to users in a meaningful and intuitive manner.

The user can have a single device mounted on a vehicle or deploy multiple devices deployed on their fleet to have access to an up-to-date insight of their roads. The data created from the incident detection can be used to automate the opening of service requests and/or work orders which helps streamline the process of resolving incidents.

A first aspect provided is a system mounted to a vehicle for identifying incidents of a roadway and transmitting the incidents to a server, the server located remotely from the system, the system comprising: a device having: a camera for obtaining digital images; at least one sensor including a location based sensor; a memory and processor for executing image processing instructions for processing the digital images for automated detection of the incidents, generating object data based on the processing, generating incident data including the object data and the images; and a network interface for sending the incident data over a communications network to the server during operation of the vehicle on the roadway.

A further aspect provided is a method for identifying incidents of a roadway and transmitting the incidents to a server, the server located remotely from the system, utilizing a memory and processor for executing instructions to: instruct a device mounted on a vehicle, the device having a camera for obtaining digital images and sensor data using at least one sensor including a location based sensor; execute image processing instructions for processing the digital images for automated detection of the incidents, generating object data based on the processing, generating incident data including the object data and the images; and send the incident data over a communications network to the server during operation of the vehicle on the roadway.

DESCRIPTION

Referring toFIGS.1and13, shown are systems10,10′ designed to automate identification and reporting of road related incidents/conditions/objects12(e.g. pavement damage, street sign damage, debris on road, etc.) with respect to a road surface14and/or adjacent road surface14surroundings13in real time. The road related incidents/conditions/objects12are hereafter referred to generically as objects or incidents12, for ease of description purposes only. Digital images16,117(referred to interchangeably as images16or images117) are taken by one or more imaging devices101, e.g. a smartphone or other portable imaging device preferably with network connection capabilities to a communications network18,121(e.g. wireless network105a, cellular network106a, etc.), referred to interchangeably as network18or network121. Only selected data portions20(e.g. image frames16a,b,c,d—seeFIG.2) of the images16can be transmitted over the network18by the device101to a remote server107,123(referred to interchangeably as server107aor remote server123) for subsequent processing/reporting, as further described below, such that image discard data19is excluded (seeFIG.4) from object data21(also referred to interchangeably as incident data120—seeFIG.13) transmitted to the server107a.

Unprocessed image(s)16can be transmitted by the device101over the network18to the server107afor subsequent processing as they are acquired in real time (subject to network18connectivity constraints) or after being temporarily stored on the device101. It is also recognized that both resultant processed image data20and unprocessed images16can be transmitted to the server107aby the device101over the network18as well together with data21that may include the sensor information17. In any event, it is recognized that it is advantageous for the systems10,10′ to be configured to preferably send acquired data21by the device101to the server107a, in order to take advantage of additional image16processing and capabilities on the server107a. Further described below (for example with reference toFIG.13) is the advantageous adjustment (e.g. augmentation) of operation of the device101in view of operator input (e.g. trigger signal124and/or voice commands/signals116) using a manual (or otherwise referred to as interactive) acquisition process803a(seeFIG.15). For example, the interactive acquisition process803acan be operated by the operator in parallel with the autonomous incident detection process307.

Referring again toFIGS.1and13, it is recognized that one advantage of the current system10,10′ is that storing and/or sending of the object data21can occur in real time while the device101continues collection and processing of additional digital images16for a second section of the roadway14different than the first section of the roadway14associated with the sent object data21. As such, the system10,10′ is configured to continuously store/send respective sets of object data21for respective differing sections of the roadway14on a real time basis and/or scheduled basis, dependent upon appropriate network18connectivity between the device101and the server107a. Accordingly, sending of the data of interest21over the network18can include queuing a set of the intended data21in memory99,104abefore transmission to the server107a. As discussed, the image data20and associated content data21and sensor data17can be affected (e.g. adjusted or otherwise augmented interactively) by the signal(s)116,124. It is recognised that the content data21can also be referred to as incident data120(e.g. containing images16,117, sensor data17,118and metadata20,119).

The device101is mounted by a mounting component103ato a body of a vehicle102a(e.g. car, truck, bus, etc.), such that an imager500(e.g. camera107,500—seeFIGS.5,13) has a viewpoint of the road surface14and optionally any desired adjacent surroundings13(e.g. roadside, curb, horizon, overhead, etc.). Accordingly, during operation of one or more vehicles102aalong the road surface14, the imager(s)500(also referred to interchangeably as camera(s)107) of the device101record/capture the images16according to the configured viewpoint (as facilitated by orienting the device101with respect to the road surface14and/or adjacent surroundings13as per the capabilities of the mounting component103a). It is recognized that the captured images16will contain recorded data of the objects12, which will be identified by image processing instructions905(stored in memory99provided by computing infrastructure100of the device101—seeFIG.7), also referred to generically as neural network(s)905.

Referring toFIGS.1and7, the system10, as an example, can utilize device101(e.g. a mobile computing device such as smartphone, smart camera or embedded computer system) physically coupled to the vehicle102a(an automobile to which the device101is attached) by the mounting component103a(i.e. attaching the device101to the vehicle102a). In general, the device101executes software108(i.e. stored instructions) which collects image data16(the digital images16) from the device's101camera(s)107,500(the imagers107,500) and infers (i.e. determines, identifies) any objects12present in the images16using the neural network(s)905(e.g. image processing instructions905) by way of a computer processor111a(also referred to as CPU106—seeFIG.13) and/or a graphics processor112a(also referred to as GPU110—seeFIG.13). The device101transmits the data20(containing information about the objects12identified from the images16) along with acquired sensor700information17,118to the server107a(via the network18utilizing the cellular connection106aor wireless connection105a, for example). The device101can store data (e.g. images16, processed data20, sensor information17) in non-volatile memory104a(of the computing infrastructure100) prior to transmitting any of the data20/sensor information17over the network18. The processed data portions20(containing the identified object(s)12of interest) and/or sensor information17can be referred to generically as object data21(e.g. seeFIG.1). As discussed herein, the embodiment of image16processing by the device101is considered only one example. As such, it is recognized that processing of images16can be performed by the device101alone, by the server107aalone, or by both the server107aand the device101as provided for inFIG.4for the shared/distributed processing environment400a.

The server107acan facilitate as gateway to users (of the device101) to make data available to other users (e.g. road supervisors, construction managers, asset managers etc.) in a meaningful and intuitive manner by way of accessing any of the data20/sensor information17received by the server107a. It is recognized that as one embodiment, the resultant processed data20containing information of the identified objects12(from the images16) can be portions of the images16themselves. Alternatively, or in addition to, the resultant processed data20can also include metadata (e.g. descriptions/descriptors) of the objects12identified from the images16by the image processing instructions905. For example, the descriptions/descriptors of the objects12can include object type (e.g. road sign, pothole, road debris, etc.), object size (e.g. 100 cm2wide by 20 cm deep), etc. In any event, it is recognized that the resultant processed data20represents only a portion of the total images16data recorded by the imager500, such that images16(and/or the descriptions/descriptors representing the images and their image contents) not containing objects12(of interest) are excluded (e.g. the image discard data19) from the resultant processed data20sent over the server107aover the network18, as one example embodiment.

It is recognized that the device101is an integral part of the system10. Referring toFIGS.1and7, the device101can be consist of a plurality of hardware and software components that are configured to automatically collect visual, location and sensor data (e.g. images16and sensor information17) while affixed to the vehicle102atravelling along the road surface14. Components that typically make up the device101are embedded in the computing system/infrastructure100, typically inclusive of the central processing unit (CPU)111aand/or the graphics processing unit (GPU)112aand memory99,104including a high speed volatile memory such as a ram as well as non-volatile memory, any of which facilitates the device101to execute its software108(e.g. operating system, image processing instructions905, etc.). The device101may also have a read only memory99for storing instructions necessary for operation of the device101. Further, the non-volatile memory104acan be associated with storing files706associated with operating system(s), component driver(s), application(s), and media, alongside other software applications resident in memory99of the device101. The device operating system (e.g. part of the software108) can be such as a windows operating system, android operating system, linux operating system, or other operating system, whether embedded or not.

The device101can also have one or more data transmitting and receiving components (communication components operating a network interface113ato the network18—also referred to as network interface122—seeFIG.13) which can be wireless. Further, the device101can interface with external wireless communication components117avia wired connector115a, such as a USB connection or an ethernet connection, in order to transmit the data20and sensor information17over the network18to the server107a, as an example. The physical wired connection115amay also be to a device which itself is wireless and transmits wirelessly to a network18.

In terms of sensors700, the device101can have, by way of example, a geo-location sensor701or a geo-positioning sensor701, provide location based information using satellite (such as GPS, GNSS, glonass, galileo) or cellular tower locations to determine device positioning information (as part of the sensor information17). The device101may also utilize sensors700related to its positioning, location and orientation such as accelerometer sensor702, gravity sensor/gyroscope sensor703, rotational axis sensor704and/or other sensors705. The device101may, in some configurations, utilize a battery307a(seeFIG.3) which can facilitate operation of the device101even when connection to vehicle power302ais temporarily interrupted. For example, in the event that the vehicle102ais temporarily shut-off, such as when filling gas, the device101can continue to operate on its battery307acharge, in order to effect the image processing of the acquired images16and subsequent sending of the resultant processed data20and sensor information17over the network18(seeFIG.4), as an example.

Further, the device101can have a user interface119aincluding a display111(seeFIG.13), built-in or external, in order to display information from the device101, to an operator of the device101, the display information such as but not limited to the camera500field of view (viewfinder), the orientation of the device101, status indicators, settings, parameters, and other information related the installation, configuration operation, and maintenance of the device101.

As further described below, the device101can execute the software108(including the artificial intelligence neural network(s)905) for detection, classification and/or feature extraction of objects12of interest in the acquired images16in order to infer (e.g. determine) what object(s)12are present in images16, the position of the object12in the images16, and other relevant information. The software108can also collect, process, store and transmit sensor information17from its sensors700for the geo physical location (e.g. GC a,b,c,d—seeFIG.2) in which the image16was acquired with respect to the road surface14(i.e. geographical coordinates of the portion of the road surface14that contains the objects(s)12).

Further to the above, it is recognized that the device101components can be packaged together in a common housing (not shown) or some components can reside externally to the device101and connected via currently common interface connector115asuch as a USB port. Examples of electronic devices that can be configured to function as the device101can include smart phones, smart cameras and embedded computer systems. For example, a smartphone101can be defined as a mobile device that contains components within to run the software108. This device101can be ideal for a portable installation facilitating for easy transfer between different vehicles102a. Example of currently available capable smartphones are Samsung galaxy s10, s10+, s20 models, Samsung note s10, s10+ models, LG g8, iPhone 11 pro, iPhone 11. It is expected that many of the newer smartphone models by the majority of the smart phone manufacturers can also perform as a device101when enabled with the software108(i.e. including the neural network(s)905as further described below).

For example, a smart-camera101can be defined as a camera with image processing capabilities (e.g. software108) that is capable to execute the instructions of the artificial intelligence neural network(s)905. For instance, a camera101containing artificial intelligence enabled chipsets/GPUs112asuch as Intel Movidius, Nvidia Cuda, Texas Instrument Sitara, Qualcomm Adreno series, etc. Alternatively, the camera101can be packaged together with an embedded CPU111a(such as the likes made by companies such as Intel, Amd or Arm) which facilitates execution of the artificial intelligence neural network905. The camera sensors700and transmission components (e.g. network interface113a) can be embedded or externally connected, as desired.

For example, the embedded computer system101can be defined as a computing device101that is designed to operate (via the software108) with some resistance to shock, vibration and temperature fluctuations. Sensors700are embedded and/or provided as peripheral (i.e. external) devices. The embedded computer system101can be considered for permanent installation in the vehicle102aor for installations with multiple cameras500. Current examples of embedded computer systems101include Nvidia Jetson AI platform, Google coral edge series, raspberry pi series, rugged industrialized embedded computers specifically environmentally hardened for use vehicles102a, or other computing devices101.

Example Embodiments for Imagers500, Also Referred to as Cameras107,500

Referring toFIG.5, the device101can be equipped with internal camera(s)501aor external camera(s)502which can be of different types such as telephoto503a, wide angle504a, or night vision505a, or other types of cameras500(e.g. imagers500), used to record a digital image16as acquired by the device101as the vehicle102atraverses along the road surface14. It is recognized that some of the images16acquired will contain object(s)12of interest, while others of the acquired images16will not contain any desired object(s)12of interest. As further discussed below, the acquired images16can be embodied as a series/plurality of image frames16a,16b,16c,16d, for example (seeFIG.2). The image frames16a,16b,16c,16dcomprise the plurality of images16acquired by the device101as the vehicle102atravels along the road surface14(seeFIG.1).

As the device101primary sensor is a camera500, the device101can include at least one camera500. Depending on the configuration of the device101, the device101can encase one or more cameras501a(for example, if the device101is a modern smartphone). The device101can also be encased in a camera501a(for example, if the device101is a smart camera, or an AI enabled camera). The camera(s)500can also be attached to the device101externally502. For example, the device101can be an embedded computer connected to an external camera via a wired or wireless interface. The device101can utilize internal camera(s)501aand external camera(s)502at the same time. For example, a smart phone101can utilize its built in camera500to process images16acquired facing the front of the vehicle102, whereas wired or wireless cameras500could be mounted on the sides and/or back of the vehicle102atransmitting images16to the smartphone101.

The device101can have different type(s) of camera(s)500. Some examples of different types of cameras500include telephoto503a, whereas the camera500is optimized to capture images in the distance; wide angle504acamera(s), whereas the camera500is optimized to detect an image16with a wide field of view; and/or a night vision505acamera, whereas the camera500is optimized to acquire images at low light settings.

Different camera(s)500can be used for different use cases. For example, a wide angle camera504amay be used to detect issues with signage12(for example, a damaged sign), whereas a telephoto503acamera can be used to detect road defects12(for example, cracks or potholes); and an infrared camera505acan be used for night time image16acquisition. The types of incidents12noted inFIG.5are for illustration and do not mean that the incidents12depicted are exclusive to the camera500type as given by example.

Further, a device101can utilize multiple cameras500, internal and/or external simultaneously. For example, the device101can be attached to a vehicle's102awindshield and have both a telephoto503aand a wide angle504acamera internally501. The telephoto503acamera may be used to detect road defects12(for example, cracks and potholes) and road issues12(for example, faded lane markings or open manholes), while at the same time, the wide angle504camera can be used to detect damaged signs12. The same device101can also be connected to two external502cameras500mounted on the vehicle's102aside windows, facing the roads and curb as part of the desired road surface14and surroundings13in the camera's field of view. The external cameras500can look for road damage12and curb damage12.

In some use cases, different cameras500can be used under different circumstances. For example, at night when lighting conditions are poor, the device101can switch from one camera500to another optimized for night vision505a. This may be done programmatically with the use of the software108, based on schedule (for example, based on a sunset timer), image parameters (such as exposure or brightness), or a sensor700(for example, when a light sensor determines it is less than certain luminance level). It can also be done manually by a driver selecting a different camera500or selecting a “night mode” which includes the night camera505a. Similarly, in some use cases, the same cameras500can be used with different camera settings. Examples of settings that may be adjusted for night time operation include frame rate, resolution, aperture, ISO, brightness, and/or shutter speed. As an example, at night when lighting conditions are poor, the device101can switch from daytime camera settings to nighttime camera settings. This may be done programmatically with the use of software108, based on schedule (for example, based on a sunset timer), image parameters (such as exposure or brightness), or a sensor (for example, when a light sensor determines it is less than certain luminance level). It may also be done manually by a driver selecting a different camera or selecting a “night mode” which includes the camera night time settings.

Some digital cameras500can give access to almost all the image16data captured by the camera500, using a raw image format. An example type of the cameras/imagers500can include digital image sensors using metal-oxide-semiconductor (MOS) technology. Another type is digital semiconductor image sensors, including the charge-coupled device (CCD) and the CMOS sensor. Another type can be the NMOS active-pixel sensor (APS). As technologies improve, more sensor types may be available to work with the cameras500.

The vehicle102auses imagers/cameras500to record the series of digital images16(as acquired by the device101) while the vehicle102atraverses along the road surface14. It is recognized that some of the images16acquired will contain object(s)12of interest, while others of the acquired images16will not contain any desired object(s)12of interest, as determined by the image processing instructions905. As further discussed below, the acquired images16can be embodied as a series/plurality of image frames16a,16b,16c,16d, for example (seeFIG.2). The image frames16a,16b,16c,16dcomprise the plurality of images16acquired by the device101as the vehicle102atravels along the road surface14(seeFIG.1). The images16can be embodied as captured individual still image frames16a,16b,16c,16dand/or embodied as captured video containing the individual frames16a,16b,16c,16d, as desired. As shown by example, image frames16band16dcontain objects12, while image frames16aand16cdo not contain any objects12. Further, it is recognized that the plurality of image frames16a,16b,16c,16dare captured sequentially with respect to one another, i.e. distributed sequentially over recorded time and distance along the road surface14, as further discussed below. As discussed, the images16may actually contain potential objects12of interest, (e.g. a pothole), however these potential objects12of interest could be processed as discarded image data19and thus not included in the resultant processed image data20(e.g. as provided by the device101and/or the server107a). It is recognised that these images16can be captured according to the various triggers, such as autonomous incident detection307, recognized voice command305, and/or trigger signal124resulting in the interactive operation803a(seeFIGS.15,16) capturing incident data120.

One example of discarding a potential object12is where a geo coordinate GC a,b,c,d matches an already transmitted object12of interest in a previous transmission object data21to the server107a. In this manner, duplication of objects12in the data20can be advantageously inhibited. Another example where a potential object12is retained in the resultant processed object data20is where the geo coordinate GC a,b,c,d does match the GC a,b,c,d of the object12in a previous transmission of object data21, however a state of the potential object12has changed (e.g. a state such as a size of the object—a pothole size of the potential object12has increased over the size of the same object12reported in previously transmitted object data21for the same pothole12at that identified GC a,b,c,d).

For example, as shown inFIG.2, the selected image frames16a,16b,16c,16dof the plurality of images16provide for some imaged portions14a,b,c,dof the road surface14or surroundings13containing objects12while others do not. As an illustrative example, the respective geo coordinates GCa,b,c,d (e.g. sensor information17a,b,c,d) for each image frame16a,b,c,dof the images16can be used to represent contiguous sections (overlapping or non overlapping in geographical coordinates) of the road surface14or surroundings13portions14a,b,c,d. As such, it is clear from the example image frames16a,b,c,dprovided that image road surface/surrounding14a(in image frame16a) and road surface/surrounding14c(in image frame16c) do not contain any objects12. On the contrary, image portion14b(in image frame16b) and image portion14d(in image frame16d) do contain objects12. Further, for example, geo coordinates GCa,b,c,d can represent physical geographical coordinates (e.g. provided by sensors700of the device101associated with at what geographical position the vehicle102awas at on the road surface14when the particular image frame16a,b,c,dwas recorded). As such, as further discussed below, the image processing instructions905are used by the device101to determine that image frames16b,dcontain object(s)12and image frames14a,cdo not contain object(s)12. Further, it is recognized that each image frame16a,b,c,dcan have a positional reference frame REFa,b,c,d, such that the physical location of the object12within the image frame16b,dcan be specified. Also as discussed below, it is a distinct advantage of the current system10to have an ability to identify those images16(e.g. image frames16a,c) that do not contain any objects12of interest and thus can be discarded and thus not be included as part of the object data21(e.g. transmitted to the server107a—seeFIG.1). As further discussed below, it is recognized that just because a particular image frame16a,b,c,dcontains an object12, as identified by the image processing instructions905, the identified object12may still subsequently be discarded by the image processing instructions905for a number of reasons (see below).

As such, any discarded image data (e.g. discarded data19—seeFIG.4)) from the images16constitutes bandwidth transmission savings for communication of the object data21between the device101and the server107a. Further, any discarded image data19(e.g. entire frames16a,b,c,dand/or selected portions of an image frame16a,b,c,d) from the transmitted object data21can also result in downstream data processing/storage savings by the server107a. It is clear, as discussed below, that the system10can take advantage of a split image processing framework400a, seeFIG.4further discussed below, such that a first portion402aof the image16processing can be performed on the device101itself (i.e. prior to transmission of the object data21to the server107a), while a second portion404aof the image16processing can be performed on the server107apost receipt of the acquisition data21. For example, the image processing instructions905can be used as a first image processing portion402ato identify objects16of interest and thus discard/exclude data19representing entire frames16a,b,c,d(or portions of frames16a,b,c,d) before transmitting the acquisition data21(image, image metadata/objects, and sensors data17). The server107acan then be tasked with identifying the relevance of the objects12of interest in the transmitted data21, including for example the physical position (e.g. REFa,b,c,d) of the objects12in the frames16a,b,c,dincluded in the object data21as part of the second image processing portion404a. It may also discard or re-acquire/refresh incident images16based on a timer that expires (for example, a certain number of days has elapsed) or based on distance from previously noted incident (as the GPS may not be 100% precise), and as such, data19can be discarded based on previously recorded incident.

Accordingly, the image discard data19is considered as those portion(s) of image16data that does/do not contain determined object(s)12of interest by the image processing instructions905, e.g. as implemented by the processor(s)111a,112aof the device101. Examples of the image discard data19are shown inFIG.6. For example, image information601a,604,605shows example image data content20as included (e.g. image data of the images16representing the identified object(s)12of interest) as well as image discard content19as discarded. As discussed, the discarded image content19can be withheld from inclusion in the object data21transmitted to the server107a(seeFIGS.1and4).

In terms of format, a digital image16can be an image16containing digital data content (e.g. both data19and data20as present prior to processing) representing picture elements, also known as pixels, each with finite, discrete quantities of numeric representation for its color intensity or gray level that is an output from its two-dimensional functions fed as input by its spatial coordinates denoted with x, y on the x-axis and y-axis, respectively (e.g. positional reference frame REFa,b,c,d). The image would be acquired either as raw image data available in various formats such as YUV RGB, HSL, HSV, or other image color spaces and encodings as available from the camera500device. The data (e.g. both data19and data20as present after to processing) may be available in the form of coordinates which can be scaled similar to vector images. Vector images can have the unique advantage over raster graphics in that the points, lines, and curves may be scaled up or down to any resolution with no aliasing. The points determine the direction of the vector path; each path may have various properties including values for stroke color, shape, curve, thickness, and fill. As such, it is recognized that part of the image16acquisition and subsequent image processing (by the image processing instructions905) can be used to incorporate data19,20which can be overlayed on the image in a vector graphics or a separate raster image and flattened/merged into the image17itself to display the detection12in the image16. The data21may be overlayed/incorporated into the image data20or stored and sent separately, with association to the respective image16. As discussed above, images16can be acquired through digital cameras500, which use any of several image file and color space formats known in the art.

It is recognized that the images16can be compressed using any known compression technology, before they are sent as part of the object data21(as discussed) to the server107a. It is also recognized that image compression is not used by the device101(e.g. by the image processing system900—seeFIG.9) to identify objects12of interest in the images16, in order to produce the processed image data20. In other words, data compression can be utilized downstream of generation of the processed image data20(used to separate or otherwise remove the image discard data19from the overall plurality of image data16acquired initially by the camera(s)500of the device101).

For example, the camera500and/or the image processing instructions905can utilize digital image compression, for those portions of the images (e.g. frames16a,b,c,ddetermined to contain object(s)12of interest), before transmitting such image data20in the data package21(e.g. seeFIG.1). For example, digital image compression technology can incorporate discrete cosine transform (DCT), a lossy compression technique. For example, as an image format for any image data of the object data21, JPEG compresses the image content down to relatively smaller file sizes. For example, the DCT compression algorithm of the JPG format can be used.

The device101is intended to be used when deployed in the vehicle102a. In many cases, the vehicle102acan be a vehicle that is operated on behalf of an organization which can be governmental, quasi-governmental or a private company. It can also be used voluntarily by individuals as a crowd-sourced application.

Examples of governmental organizations include all levels of government, including national, federal or republic government; provincial, territorial or state government; municipal government, including municipalities, upper tier municipalities (such as metropolitan, regional, county or other name used to describe major upper tier municipalities), or lower tier municipalities (such as city, village, township, town, community, or other name used to describe). The governmental organization may also be a special organization, such as a reserve, resort, other name that is used to describe the local government of a certain geography and population. Examples of quasi-governmental organizations would be government-owned or supported organizations. Those could be organizations established as part of a public-private-partnership or a concession to build, maintain and/or operate an asset or a service over a period of time. They could be separately incorporated but the government may have full ownership, majority ownership, or minority ownership. The government's representatives can sit on the board of such organizations. Examples of quasi-governmental organizations include toll road concession companies, bridge concession companies, transportation and/or transit authorities, and/or utility or telecom companies. A private company can simply be a private company that is the owner of the asset that is to be inspected, or contracted on behalf of the owner to do so.

The vehicle102acan be a service vehicle dedicated to patrolling an area for the specific purpose of identifying incidents/objects12on behalf of the organization. The vehicle102acan be a car, a truck, a golf cart, a tractor, an atv, a bicycle, an e-bike, a motorbike, a motorcycle, a snowmobile, a van, or a customized utility vehicle, for example. The vehicle's102aprimary purpose can be different than acquiring incidents/objects12, but augmented with the device101mounted for a supplemental function of incident/object12detection on behalf of the organization. Examples include garbage trucks, snowplows, operational service vehicles, utility vehicles, road and sidewalk sweepers, public transportation vehicles such as buses, school buses, and transportation vans or taxis. The vehicle102acan also be a private vehicle owned by an individual, whereby the individual contributes incident object12data detected by the device101on a good will to the organization, or for monetary compensation. The vehicle102acan also be an autonomous vehicle102aoperated privately or by an organization, whereas the autonomous vehicle102acan be equipped with the device101to automatically detect incidents/objects12in the area in which it operates.

The device101is intended to be mounted in the vehicle102ausing the mounting component103a. Typically, the device101can be mounted to the vehicle's102awindshield or a portion of the body, though it can also be attached to the dashboard, the side windows, the back windows, or the vehicle102aframe, as desired. The mounting component103aconfiguration can be different for different vehicle102a/device101combinations. For example, depending on whether the device101is a smartphone, a smart camera, or an embedded computer with external camera different mounting configurations can be used. Different mounting configurations can also be used depending on whether the device101is to be permanently affixed to the vehicle102or transferable between different vehicles102a. The vehicle102acan utilize the mounting component103aof different types. For example, the mounting component103acan be attached to the vehicle102avia a suction cup, a sticky tape, glue, or screws and bolts, for example. The mounting component103acan allow for an easy removal of the device101by having the device101easily detach from the mounting component103a. The mounting component103aitself can also have multiple parts which facilitate detaching parts of the mounting component103atogether with the device101.

Non Volatile Memory104

The storage capabilities (e.g. memory99,104, etc.) of the device101can have the non-volatile memory104associated for storing files706associated with operating system(s), component driver(s), application(s), and media, alongside other files used in software applications. The non-volatile memory104can be embedded in the device101, an add-on, and/or a peripheral. For example, smartphones currently come with built in non-volatile memory104, which can be expanded using non-volatile memory104add-ons (for example, micro-sd memory). Embedded computers typically come with a variety of hard drives, flash drives, and interfaces which allow for including one or more non-volatile memory104storage components.

The device's101software108can store digital data, such as the acquired incident images16, data portions20of the images16including data about the identified objects12from the images16, and associated sensor700data (e.g. sensor information17) onto the non-volatile memory104. It is recognized that the data portions20are those determined excerpts (e.g. frames16a,b,c,d) from the total images16that contain the objects12of interest.

The device101(e.g. via software108) can store the images/data21to the non-volatile memory104prior to transmitting over the network18for a variety of reasons. For example, cellular106aconnectivity may not be reliable, and the data21would have to be stored temporarily before transmitted. If cellular connection106aor wireless connection105ato the internet18is not available (e.g. out of cellular network coverage or out of the wireless range) during operation, the software108will store the images16,20and their associated data17to the device's101non volatile memory104. Once the connectivity has been restored, the software108will resume uploading the object data21to the server107a.

Some organizations may also opt to utilize wireless connectivity105ain order to save on cellular data costs, and as such, the data21would be stored until such time that the device101has access to wireless105aaccess point, which is connected to the network/internet18. The device101may also benefit from a performance, power, and/or heat management perspective to only transmit data once the vehicle102ais idle (for example, idle in traffic or in a parking lot). The device101may also be benefit from a performance, power, and/or heat management perspective to initiate an upload process as a scheduled process as opposed to an ongoing process.

The files706and/or database707(representing any of the images16, sensor information17, and/or data portions20—i.e. processed images16) may be in an encrypted format in the memory104to secure the information. The files706and/or database707are deleted from the non volatile memory104when they are successfully uploaded to the sever107a. The data16,17,20can be stored on the non volatile memory104in various file formats. Image data can be stored as compressed or uncompressed files such as jpegs, bitmaps, webp, png, and other common image formats. The associated sensor data17can be stored as a metadata file (for example, an xml file), a table file (i.e. a csv or a txt file), or in the database707. The files706may also be stored in a common file706format or in a proprietary one.

The device101can include the wireless connection105anetwork interface in order to connect to the server107a. For example, a wireless connection105amay be wireless Ian or wi-fi, operating at common frequencies such as 2.4 ghz, 5 ghz, or other frequencies typically associated with ieee 802.11 or other wireless standards. The wireless connection105ahas the advantage that it typically allows to upload a large volume of data without the cost of cellular data usage.

The device101can include a cellular connection106anetwork interface in order to connect to the server107a. For example, the cellular connection106acan utilize technologies such as 3g, 4g, lte, 5g or other technologies used for access to cellular towers. The cellular connection106aprovides for communications with the server107aon a constant, frequent or periodic basis, allowing the incident information/asset data12(contained in the object data21) to be generated as communications are taking place. It is expected to be used in many scenarios including when cellular cost is not a major issue, when cellular connection is available, when faster response to incidents is necessary, and/or when the device101does not have access to wireless connectivity105a.

The server107ais responsible for the organizing, storing, processing and disseminating of the object data21uploaded by the device(s)101. A single server107acan host a plurality of users, whether governmental users, quasi-governmental users, or private organizations. A single server107acan communicate with a plurality of devices101as clients of the server107a. The object data21of each client is securely segregated from each other where one user cannot access the object data21of other users, unless a user purposely marked their data for sharing, for example. When the server107areceives object data21from a device101through the internet18, the image data (e.g. frames16a,b,c,d) containing objects12of interests are stored in a server storage30to a folder which may be allocated specifically to the particular device101and user's organization. It can be also organized by date, road segments, or other hierarchal structure, as desired. The server107acan store image data16, sensor data17, object data21, resultant processed data20, resultant processed data20′ and/or discard data19,19′, as desired. For example, the device101can send the discard data19to the server107a, as part of the object data21or subsequent to sending the object data21, as desired.

Accordingly the discard data19,19′ can be defined as the data removed from the images16in order to produce the object data21(e.g. containing the processed images20). For example, the discard data19could be inclusive of one or more whole frames/images16a,b,c,dthat are dropped (e.g. removed from the images16and/or sensor data17) before sending the resultant object data21(as a result of utilizing the image processing instructions905on the captured images16and/or sensor data17). For example, the image processing instructions905can also include utilizing the sensor data17in order to identify the objects12and/or discard data19. For example, the sensor data17(such as direction and/or GPS data) can be used by the device101to recognize and extract the discard data19from the image data16. For example, images16captured are associated with sensor data17during their capture (e.g. direction of vehicle travel and GPS data for recognizing the geo position of the captured images16). As such, when new subsequent images16are taken, having sensor data17(e.g. direction and/or GPS data) matching that sensor data17of the previously recorded images16, the newly acquired images16can be discarded (either in whole or in part). The newly acquired images16could be considered by the device101(as a result of executing the image processing instructions905on the previous images16and newly acquired images16and their associated sensor data17) as duplicate data in view of the match in sensor data17between the previous images16and the newly acquired images16.

Additional information, such as sensor data17, geographical coordinates GCa,b,c,d, direction of travel, date/time, pitch, and/or data of the object data21describing the object12of interest in the image can be associated to each incident image (e.g. image frame16a,b,c,d) and also stored into one or more database(s)30on the server107a. The server107ais depicted in the system10,10′ as a single server107a. While it may be deployed as a physical server, there may be more than one server107asegmented by geography (for example, Canada, US, Mexico or other countries), by architectural function (DNS, runtime, database, storage, image processing, reporting, or other function), by capacity (for example, users 1-1000 reside on server 1, whereas users 1001-200 reside on server 2), logically (such as a virtual machine that runs on a server cluster, or on a cloud), or in other common ways in which servers which provide software as a service are setup. For greater clarity, the word server107aand server(s)107awill be used interchangeably throughout the description, figures and claims as the system10,10′ could be setup to use one or more physical, virtual or cloud based servers107a.

The software108can be responsible for acquiring image16data from the camera500, acquiring geographical positioning data17from the geo-positioning sensor701, collecting sensor(s) data17, collecting other system information such as date and time, identifying incidents12and assets data12using image processing function(s) and neural network(s) inference workflow(s) of the image processing instructions905, and reporting the incident/asset data12(considered of interest) through communications of the object data21with the server(s)107, for example. The software108is responsible for other functions, which are relevant to the operation of the unit, such as storing images and data to the device's101non volatile memory104in an encrypted or not encrypted, redacted or not redacted format, and controlling the content on the device's101user interface119a, if available. The software108may also provide functions pertaining to the configuration, calibration, and operation of the device101. The software108may also communicate with the server(s)107afor the purpose of downloading updates, settings, configuration parameter(s), neural networks(s), data and/or files. The software108may also communicate to the server(s)107aof sending non-incident information, pertaining to the performance, status or diagnostic information for the device101.

Referring toFIG.3, shown is a variety of options which the device101can be powered. A device101such as smartphone, smart camera or embedded computer system, with an optional battery301athat is connected through a power connector305ato a vehicle battery307aor to a vehicle power supply302athrough a fuse panel303aor auxiliary port(s)304adirectly or through a voltage converter306a. The device101may have an optional ignition sense308aport to know when to the device101on or off as to not drain its own battery301aand/or the vehicle battery307a. The device101, may have a battery301athat allows it to operate for a period or to keep it in a suspended state for a period of time without a direct connection to power. For example, nearly all smart phones today come equipped with a battery301a.

The device101typically connects to the vehicle's102apower supply302athrough the vehicle's102afuse panel303aor through auxiliary port(s)304awhich are typically known as automobile auxiliary power outlet, car outlet, automotive power socket, automobile outlet, or vehicular outlet. The auxiliary port(s)304amay be with a USB connector, a plug connector, a socket connector, or other commonly used connectors. A vehicle's102apower supply302avoltage typically ranges from 9v to 36v. The device101may be a smart phone, a smart camera, or an embedded computer with varying power requirements. Depending on the compatibility between the device's101power requirements and the vehicle's102apower supply302avoltage, a voltage converter306amay be required. The power load supplied by the vehicle's102electrical system would typically allow the device101to operate and charge its battery301a(if exists) simultaneously.

Depending on the configuration of the vehicle102a, when it is idle or turned off, power available to the auxiliary port(s)304amay be shut off to protect the vehicle102abattery from being drained. The device101can be connected to the vehicle102apower through a power connector305awith a compatible header. For example, if it is a smartphone the power connector305amay be a USB-c, a lightning plug, or another charging cable variant. The other side of the power connector305amay be a USB, a socket plug, a wire harness, or other connector that is aimed to connect the device101to the power supply302athrough the power system. The device101may also utilize proprietary or standard cables for power connector305afor other device101variants. In the event the device101is an embedded computer, it may be connected directly or indirectly to the vehicle's battery307a. It may also have an ignition sense308ainterface that will communicate to the device101when to turn off or on as to not to drain the vehicle's battery307a.

Privacy & Security

Referring toFIG.6, shown is a privacy/security framework600including capabilities provided by the image processing instructions905for the redaction (e.g. generation of discard data19,19′) of personally identifiable information601asuch as faces602a, license plates603a, or other objects12containing personally identifiable information604such as cars or people. In certain circumstances the framework600has the option to redact whole image605other than objects12of interest (also referred to as the resultant image data content20), depending upon a defined privacy criteria. The framework600has the option to utilize unredacted images606, with processing on device101and/or server107a. The system may use secure encrypted image607storage104on the device101and/or server107a. It may also utilize secure encrypted607communications to the server107a. It is recognized that the images16once processed (examples as processed image results601a,602a,603a,604,605,606) can be referred to as the processed image portions20(i.e. such that the data content of the processed image content20does not contain the discard image data19,19′).

For example, discard data19,19′ can include data such as but not limited to: portions of images16which are being blurred; images16which are determined to not contain incidents12; and/or images with incidents12which are being discarded due to their position (i.e. duplication in relation to an overlapping incident12as discussed above with respect to previous images16and newly acquired images16).

As such, the image portions20contain less image data content than the unprocessed images themselves16(as acquired by the imager500). It is the image portions20(the result of processed images16as performed by the image processing instructions905) themselves that can be included as processed image data20in the object data21transmitted to the server107aover the network18, for example. It is recognised that image16portions can refer to inter frames (i.e. some whole frames that are retained/dropped) or intra frames (i.e. area(s) within image16that are retained/dropped). The privacy/security framework600is one example of how the data of the images16can be reduced (by identifying and thus extracting the discard data19,19′) by the image processing instructions905before the object data21containing the objects12of interest are communicated to the server107aor otherwise stored as processed image data20′ in the storage30by the server107a. It is recognized that an object12of interest included in the object data21can also be in redacted form (e.g. blurred out or pixel substituted), as such the included object12of interest in the resultant object data20can also include a reduction in its data size. It is recognised that pixel blurring can result in an increase in data size of the object data21(as compared to the same images16containing non blurred content). It is also recognised that pixel substitution/deletion can result in a decrease in data size of the object data21(as compared to the same images16containing non substituted/deleted content). Further, it is recognized that a particular image data16(e.g. image frame16a,b,c,d—seeFIG.2) can include redaction data19,19′ (e.g. substituted or blurred image frame portion(s)) as well as object(s)12of interest. Accordingly, any particular image data16, prior to processing, can include only “to be determined” image discard data19,19′ only “to be determined” object(s)12of interest, or both “to be determined” image discard data19,19′ and “to be determined” object(s)12of interest. It is recognized that “to be determined” can also be referred to as “potential” or “candidate”, as desired.

Considering that the system10is intended to be operated in the public space, and it is expected to be used extensively in public spaces and by governments and quasi-government organizations, the system10can have one or more privacy and security options intended to address regulations or guidelines regarding the collection and storage of data including personally identifiable information (pii) in the images16. In relation to data, pii may be pictures which uniquely identify individuals. Examples of personally identifiable information include images of people's faces602or of a vehicles' license plate603a. It may also include house addresses. Sometimes, the objects12themselves may identify an individual, for example a fairly unique car or house. Governmental organizations typically have regulations and legislations related to the handling and storage of pii. Images16acquired by governments are also typically subject to freedom of information requests. As such, in many occasions governments do not want to store pii. Governments and organizations responsible for maintaining assets are also typically subject to litigation and claims related to incidents causing property damage, personal injury, and/or death. Images12which are acquired by device101may occasionally be used as evidence against the system user in claims. As such, some users would only wish to retain the incident12data, but not any other information. For example, an image16may report an incident12of a pothole, but in the image16peripheral there may be a broken sign12not detected by the system. Such image16can then be used as an evidence against the owner of the information in claims. The system10can store an acquired image16containing as an incident12as an unredacted image606containing all of the images'16original information. However, the device's101software108can have options built in to exclude pii and other non-detected related data, using the privacy/security framework600as discussed above.

Users opting to redact information may use the image processing instructions905to redact the whole image605(e.g. image16) other than the detected object(s)12of interest. The object(s)12of interest can be maintained in the picture16, whereas the remainder of the picture16can undergo image processing intended to redact the image16in order to generate the resultant processed image data20,20′ and the discard data19,19′. The object(s)12of interest, can result in image data20,20′ in which all image data16is redacted other than the object(s)12of interest. For example, a pothole12may be an object12of interest identified by the system10. After the image16redaction process, seeFIG.9by example implemented as system900, the only object12that is not redacted in the resulting image data20,20′ can be the pothole12itself.

Users may also use the image processing instructions905to redact pii using object redaction604feature aimed to redact potential objects12containing pii, such as vehicles, cars and people. It may also only blur objects which are pii, such as license plates603aand faces602a. In this instance, only the pii objects601aand/or objects containing pii604will be redacted (included in the discard data19) whereas the rest of the image16will remain untouched and thus such content being used as the resultant processed image data20,20′.

The redaction operation of the image processing instructions905could obfuscate the details in the image16parts which are to be redacted (i.e. included in the discard data19,19′). Examples of redaction operations used to generate the discard data19can be pixel substitution and/or blurring. Pixel substitution is a process where pixels occupied by the object12of interest (whether such boundaries are semantic segmentation instances or bounding boxes) are replaced by pixels of a single color or a pattern of colors, or otherwise treated as an absence of image data20,20′. A blur is a visual effect function that makes the details in resultant image data20appear fuzzy or out of focus. The redaction operation can take place on the device101or on the server107a, depending on whether the first image processing portion402aor the second processing image portion404ais utilized (seeFIG.4) for the particular redaction operation.

For example, referring toFIG.4, the first image processing portion402acan be used to generate discard data19(using pixel substitution) and the resultant image data20(that includes object(s)12of interest), such that the resultant image data20is included in the object data21and transmitted to the server107a. Once received, the server107acan use a set of image processing instructions905to implement pixel blurring and/or additional pixel substitution, thus generating further discard data19′ as extracted from the resultant image data20of the object data21. The server107awould then store modified object data21′ in storage30, which includes the resultant image data20other than the extracted discard data19′ (seeFIG.1).

In the event that the resultant images601a,604,605,606and their associated data are stored on the internal non volatile memory104, the resultant images601a,604,605,606and data may be encrypted using a modern encryption algorithm by the image processing instructions905, which would obfuscate the files706. Therefore, in the event that the device101is stolen from a vehicle102aor lost, the information (e.g. object data21) stored on the device101would not be easily accessible. Communication of the object data21to the server107amay place over encrypted communications to ensure that the data is secure in transit. Finally, stored information (e.g. object data21) may be encrypted on the server107ato ensure that the data is secure while at rest.

Referring toFIG.7, the device101can have a variety of sensors700. In addition to camera(s)500, the device101is equipped with a geo-positioning701sensor. The device101may also include an accelerometer702sensor, gyroscope703sensor, rotational vector704sensor, and other sensors705that can provide information17regarding the movement, position and/or orientation of the device101and/or the vehicle102aon which it is equipped. The sensor(s)700and camera(s)500data17,16is processed by the software108(including the image processing instructions905) before being sent to the server107a, as discussed herein. The sensor data17and image data16,20may be stored on the device101non volatile memory104in the form of file(s)706or database707entries prior to being transmitted to the server107aas object data21.

The device101includes a geo-positioning sensor701to determine its geo-spatial coordinates17. Geo-location, or a geo-positioning sensor701, provide location based information17using satellite (such as GPS, GNSS, glonass, galileo) or cellular tower locations to determine device positioning information, which is associated with the images16(e.g. geo coordinates GCa,b,c,d—seeFIG.2). In addition, the device101in many instances will have additional sensors700. For example, modern smartphones101on the market today have a variety of sensors700embedded right onto them, which provide information that can be used to determine the device101orientation, pitch, magnetic pole direction, geo-spatial position, velocity, acceleration, shock, vibration and other data17related to position and movement. The device101may include an accelerometer sensor702used to measure the acceleration force17applied to the device101across its x axis, y axis and z axis. The force may or may not include the force of gravity. The acceleration data17will typically be available as meters per second squared (m/s2) though it may be in other units (for example voltage) that can be converted to such units. The device101may include a gyroscope sensor703used to measure the rate of rotation across the device's101x axis, y axis and z axis. The gyroscope data17will typically be available as radians per second (rad/s) though it may be in other units (for example voltage or frequency) that can be converted to such units. The device101may include a rotational vector704sensor used to measure the degree of rotation across the device's101x axis, y axis, z axis and an optional scalar product or quaternion. The rotational vector data17will typically be degrees, though it may be in other units (for example voltage) that can be converted to such units. The device101may include other sensor(s)705to measure a variety of other conditions17related to the movement, acceleration, forces applied and position of the device101. For example, the device101may include a gravity sensor700, which would measure the force of gravity17in relation to the device101. Other sensor(s)700may also be magnetometer which can determine the device's101position17in relation to the magnetic north or true north. Other sensor(s)705may also include hardware and/or software monitoring of the device101components. Examples of sensors700can include battery level sensor, battery temperature sensor, CPU temperature sensor, GPU temperature sensor, ambient temperature sensor, CPU core utilization, CPU overall utilization, luminance sensor, proximity sensor, and other built in sensors available for the device101.

Any and all of the above discussed sensor type data (i.e. sensor data17) can then be associated with camera(s)500images in order to determine additional insights. For example, the sensor data17may be used to derive ridership experience, level of vibration, the speed in which the vehicle102ais travelling, whether the device101is within a geo-zone, or the estimated geo-positioning of an object12detected in an image16in relation to the device101. The sensor data17may also be used to optimize the performance of the device101in relation to the current heat, power and processing situation.

The sensor(s)700and camera(s)500provide for data17,16to be acquired and processed by the software108. The resultant processed data16,17(e.g. using the first data processing portion402a) is then either transmitted to the server107aor stored on the device101non volatile memory104until transmission can take place. The data16,17,20may be stored as file(s)706in variety of formats, such as xml, csv, txt, or in a proprietary format. The data16,17,20may also be stored in a database707. The data17,20may be stored and transmitted in encrypted on non-encrypted format.

The data17,20may be further processed on the server107ausing the second data processing portion404a. For example, it may be correlated with road segments, assets, and other information to derive additional insights. For example, what roads or assets were inspected. It may also be used by the server107for detecting alerts related to device constraints (heat, power, or processing capabilities) on the device101.

The GPS/GNSS receiver (i.e. position sensor701) can be used to record (sensor data17) the location coordinates GC a,b,c,d where the incident12occurred so that the location of the incident12can be presented. GPS can also be used by the instructions905to determine the speed the vehicle102ais travelling which is used to activate speed enabled features within the application (e.g. software108features of screen lock, driver attention warning). In addition, the GPS data17can be used as breadcrumbs' to track the road surfaces14that have already been inspected, thus when the instructions905are used to compare the geo coordinates GC of the newly acquired images16, those images16being determined as duplicates (i.e. having matching geo coordinates GC to previously acquired images16) can be discarded/excluded from the object data21. Furthermore, GPS data17can used by the software108to determine if roads14/surroundings13being scanned are within a defined geofence zone, otherwise detections12will be ignored.

The accelerometer and magnetometer sensors700can be used by the software108to determine the vehicle's102adirection of travel. Being able to determine when vehicle102adirection of travel is necessary for the system10to present which side of the road (e.g. North, east, west or south bound lane) is being scanned by the system10. It can also help to identify the incident12location in relation to the device101or the vehicle102a.

In view of the above, the rotational vector sensor704can be used to determine the device101orientation, including pitch, facing direction, vibration and bumps and such information17is sent to the server107together with the image data20. Further, it is envisioned that the device101can keep track of rotational vector sensor704information17for the purpose of integrating the data with data obtained from images16for the purpose of detecting road quality and road roughness levels as determined by the level of “vibration” or “bumps” detected by the sensor704, and such data is sent to the server107afor the purpose of being correlated to the image data20uploaded.

The accelerometer702sensor can determine the device's101acceleration force along its x axis, y axis, z axis and such information17is sent to the server107atogether with the image data20. The gyroscope sensor703sensor can determine the device101orientation17, and such information17is sent to the server107atogether with the image data20. The other sensors705can be such as magnetometer705sensor to determine the device101orientation17, and such information is17sent to the server107atogether with the image data20. The device101can keep track of the location, via the sensor(s)700, in which the device101was present through gps breadcrumbs or routes as evidence that the device101inspected the area.

Referring toFIGS.4and8, one example of a distributed processing system400ais a first image processing portion402afor image pre-processing, which is the process of manipulating and/or adjusting the image16data acquired by the camera500to provide that the data received for neural network905processing is inputted in a way which will optimize the results and performance of the system10,10′ (i.e. optimize the quality and quantity of the resultant image data20in the object data21).

For example, using the software108, a resolution for the image16data can be selected from one of the camera's supported resolutions801. The camera resolutions801can be represented as a name, such as 8k, 4k, 1080p, 720p, hd, or other common names. It can also be represented as a resolution, representing the number of pixels and typically in a format of width×height, for example 7680×4320, 3840×2160, 1920×1080, 1270×720, or other resolutions. In many instances, neural networks905can be optimized to accept images16in certain resolution, typically referred to as “input shape”. For example, an image16can be acquired by the camera500at a resolution of 1080p (1920 pixels×1080 pixels). However, the neural network905model can be trained on images16scaled down to the size of 300 pixels×300 pixels. The software108then needs to resize, or adjust the resolution801of the image16dimensions from 1920 pixels×1080 pixels to 300 pixels×300 pixels in order for the neural network905to process it appropriately. The neural network905model can have a different input shape than 300 pixels×300 pixels it may be higher (for example, 600 pixels×600 pixels) or smaller (for example, 224 pixels×224 pixels). Typically, the larger the model “input shape” or resolution, the slower the images16will be processed, however the larger the “input shape” resolution is, the more details will be retained in the image16which may increase the model's effective detection parameters, such as accuracy, recall, precision, f-score and other such metrics.

The software108can also facilitate for a field of view adjustment802, the field of view adjustment802can be optical zoom level adjustment, if supported by the peripheral camera500of an embedded computer system101. If further magnification is required to calibrate the camera's500field of view, the digital zoom level can be adjusted through the software108to achieve the desired optimal field of view. The software108may also select from a variety of internal camera(s)501aor external camera(s)502in order to adjust the field of view802. Different field of view may be optimal for different use cases on vehicle(s)102a. For example, the height and pitch in which the camera500is mounted may be different for a bus, a service truck, or a sedan, and may require different zoom levels in order to cover the same number of lanes. Similarly, different field of views may be preferred for different objects12. For example, signs12may favor a wider field of view that covers the surroundings13whereas road defects12may favor a narrower field of view covering the road surface14.

The software108can also facilitate for cropping803parts of the image16. Cropping the image16allows to omit areas which typically do not require neural network905processing. For example, if the camera500is mounted on a windshield, the top 20% of the image16may typically be sky, whereas the bottom 10% of the image16may be a dashboard or a hood of a vehicle102a. By cropping parts of the image16which are not relevant, those areas are less likely to generate false detections12or false incidents12. In addition, in the event that the neural network905“input shape” is lower than the acquisition resolution, by cropping out irrelevant portions of the image16, less detail is lost in the resizing operation of the image16. Cropping803may also be used for extracting a portion of an image16for additional image processing and/or inference activities905. For example, a car102amay be detected12by a neural network905and then cropped from the image16. The cropped car102amay then be either redacted604or processed through a neural network905that is trained to identify license plates in a picture16. Operations related to redaction such as personally identifiable information redaction602a,603aobject redaction604and/or image redaction605are also considered image processing operations as performed by the software108and related instructions905.

In the event that the image16data acquired by the device's101camera(s)500is in a format that is not compatible or optimized to be used with the neural network905architecture or library, color space conversion804may be required. Examples of color spaces include yuv, rgb, hsv, hsl, cmyk and others. Even within color spaces, there are variations in the container, structure, channels, order, format, decimal system which may require conversion. For example, a file may be represented in bytes, words, or hexadecimal. Another example is that rgb channels may be ordered as bgr. Another example is that extra channel may be present to represent transparency (rgba).

Once the image data16is preprocessed by the first image processing portion402aby the device101, e.g. using image processing instructions905, then the resultant object data21would be sent to the server107afor implementation by the server107aof image processing instructions905during a second image processing portion404by the server107a, which then results in the resultant object data20′ being stored in the storage30(seeFIG.1). As such, it is recognized that the device101can have the image processing instructions905as an embodiment of the system10,10′. As such, it is recognized that the server107acan have the image processing instructions905as an embodiment of the system10,10′. As such, it is recognized that both the server107aand the device101can have the image processing instructions905as an embodiment of the system10,10′. The device101and the server107acan have different variations, configurations, and parameters relating to the image processing instructions and neural network(s)905used.

Referring toFIGS.1and9, shown is an example of image processing system900implemented by the software108(including the image processing instructions905) on the acquired images16, in order to produce the processed image data20to be included with the object data21transmitted to the server107a. It is recognised that these processing instructions905of the system900are used in the regular AI process301—seeFIGS.15,16. It is also recognized that the example of image processing system900implemented by the software108(including the image processing instructions905) on the resultant image data20, in order to produce the processed image data20′ to be included with the received sensor information17stored in the storage by the server107a.

The software108can include image instructions905(e.g. including artificial intelligence neural networks905), for image16processing and inference for flexible workflows906inclusive of neural network(s)905inference operations907including detection902, classification903, and segmentation904, in order to generate the discard data19,19′ as well as the resultant processed image data20,20′. It is recognized that the workflows906can include a plurality of different numbers/combinations of the operations907in any order, as configured in the image processing instructions905, in order to identify, classify and segment any object(s)12in the image(s)16under consideration. The system900also depicts processing800of images16containing object(s)12of interest in relation to incidents. One image16may have several different workflows906applied to it. The object(s)12of interest are also referred to as classes12. The class12refers to one of the output categories for the object(s)12of interest. For example, they may include but are not limited to: pothole12, car12, person12, sign12, etc. The network(s)905can detect, classify, and/or segment one or more classes12(also referred to as object(s)12of interest) in the image16.

It is recognized that the identified object(s)12of interest are included in the processed image data20while the discard data19is excluded from the processed image data20, as one embodiment, such that the processed image data20and the sensor data17is transmitted to the server107aas object data21.

Further, it is recognized that the identified object(s)12of interest are included in the processed image data20′ while the discard data19′ is excluded from the processed image data20′, as one embodiment as implemented by the server107ausing the object data21obtained from the device101.

Further, it is recognized that the identified object(s)12of interest and discard data19are included in unprocessed images16sent to the server107aby the device101as the object data21(including the sensor data17). Once received, then the server107awould then process the images16as processed image data20while the discard data19is excluded from the processed image data20, as one embodiment as implemented by the server107ausing the object data21obtained from the device101.

Typically, image(s)16acquired by the device's101camera(s)500are available in some initial resolution, color space, and formatting. It is expected that in many cases, the image(s)16may need to undergo image processing800operations to optimize their compatibility with the neural networks905used and the object(s)12of interest which they are trained to identify. Some examples of image processing800operations are resizing or adjusting resolution801, field of view adjustments802, cropping803, and/or color space conversion804, as shown inFIG.8by example only.

As such, the image processing800operations can include the resolution of the image16can be set based on the available resolutions present on the camera500device, whether available as resolutions or as a name representative of the resolution. Further, the field of view can be adjusted via adjusting the optical zoom levels of the camera(s)500. Further, the field of view can be adjusted by a digital zoom process, wherein the picture16is magnified and only the parts of the image16that remain within the original dimensions are processed. Further, the region of interest12in the image16can be set. Once set, the region of interest12will be cropped. Further, the image processing can include color space conversion, whether from one space to another, or adjusting the formatting, order and/or channels of the utilized color space.

For example, the processing instructions905(e.g. neural network905) can be defined as a set of functions, operations and/or instructions which facilitates for the system900to train itself based on annotated datasets, commonly referred to as “ground truth”. Once trained, the system900can then infer on new datasets. The process is known as machine learning. The neural network(s)905utilized in the system900can be primarily geared towards identifying object(s)12of interest in images16for automated incident identification and reporting. Once processed using the image processing instructions905, the system900outputs the processed object data20, shown by example inFIG.6. Further, software108is configured to construct the object data21by associating the sensor information17(e.g. including geo coordinate data GC a,b,c,d) for each of the objects12of interest. It is recognised that during the processing of the images16using the image processing instructions905, some of the image16data acquired will be discarded in view of the discarded image16data does not contain any objects12of interest as determined by the image processing instructions905as executed by the processor111aand/or GPUa112of the device101(seeFIG.7). It is recognized that discarded image16data can be referred to as discarded data19(seeFIG.4), such that discarded data19is not included in the object data21and is thus can be inhibited from being transmitted over the network18to the server107a.

The neural network(s)905utilized can have a plurality of architectures which pass the image16through a sequence of layers operations907which are aimed at aggregating, generalizing, manipulating and/or modifying the information of another layer for the purpose of inferring, detecting, classifying and/or segmenting objects12in images16. Examples of some typical operations in neural network(s)905are: (a) convolution; (b) rectification; (c) fully connected; (d) pooling layer (e) bottleneck and/or (f) loss layer.

The architecture of the system900can be a neural network905architecture such as: (a) single shot detector (ssd), (b) you only look once (yolo), (c) convolutional neural network (cnn), (d) region-based convolutional neural network (rcnn), (e) fast region-based convolutional neural network (fast rcnn), (d) faster region-based convolutional neural network (faster rcnn), (e), mask region-based convolutional neural network (mask-rcnn), (f) region-based fully convolutional networks (r-fcn), or other published or proprietary neural network905architectures.

When a neural network905is trained on an image16set (e.g. a series of image frames16a,b,c,d), it can set certain parameters commonly known as weights. The parameters, or weights, are typically stored in a model file, or weights file. The neural network905utilized in the system900can be trained using published, well known, weights files as the basis. For example, mobilenet (such as mobilenetv1, mobilenetv2, mobilenet v3), inception (such as inception v1, inception v2, inception v3), vgg, or other popular pre-trained networks, and can be composed of different number of layers (for example, resnet50, resnet101). However, the concept of such pre-trained neural networks905is the same whereas a base architecture with base weights is modified whereby one or more of the last or final layers is modified to detect or classify a set of objects12of interest, which may be identical, exclusive, partially inclusive, or fully inclusive of the original trained objects and may include new objects not present in the original neural network905. Neural network(s)905may also be of a proprietary custom architecture with weights or parameters which are trained from scratch.

The neural network(s)905may be utilized as a detector902, seeFIG.9. A detector902typically identifies an object12of interest in image(s)16, and the location of the object12. The location of the object12is typically in the form of a bounding box represented by coordinate(s) REFa,b,c,d and/or distance(s) in relation to a point of reference902in the image16, seeFIG.2. A detector902may also provide a score, typically known as confidence, which represents how sure the neural network905is in the object12detection. A detector902may also detect landmarks. Landmarks are points of reference in a known object12. For example, in the context of a detector identifying a sign, the bottom of the sign pole and the top of the sign pole may be landmarks. Such landmarks can then be analyzed to derive further information about the status of a sign—for example, whether it is crooked or not.

The neural network(s)905can be utilized as a classifier903. A classifier903has a list of potential classes, or object types, which it is trained to identify in a picture. When processing image(s)16, a classifier903typically returns a list of potential object(s)12in the image16, sorted by the model's confidence of their presence in the image16. The neural network(s)905can be utilized as a segmentor904. A segmentor904typically segments image(s)16into regions. The regions are then typically predicted to belong to a certain class12, or type, which allows to extract a mask, or a pixel blob, that represents the class12. A segmentor904can also separate instances of the object(s)12into separate object(s)12representing one or more classes12. For example, a segmentor904may identify a pothole12, and also the shape of the pothole12, which will allow to estimate its surface area and severity.

The neural network(s)905can be designed and/or optimized to be used on the device's101gpu, cpu or both. The workflows906may utilize one or more neural network(s)905, and the neural network(s)905may be used in a sequence. One neural network(s)905can responsible for detecting902objects and/or regions of interest in the image(s)16, and one or more additional neural network(s)905can be responsible for classifying903the objects12and/or regions of interest already detected in the image(s)16. For example, a neural network905may detect902a pavement crack12, crop it with image processing800, and then another neural network905classifies903it as a longitudinal type of crack12. It could also be used to verify that the first detection is correct. For example, the first neural network905may detect902a pothole12, crop it using image processing800, and pass it to a classifier903which confirms it is a pothole12and not a manhole. In some situations, this process provides the opportunity to classify903the object12of interest using a higher resolution, since the detector902may infer on a scaled down version of the image16, whereas the cropped image16would be inferred at a higher resolution.

One neural network905can be responsible for detecting902objects12and/or regions12of interest in the image(s)16, and one or more additional neural network(s)905is responsible for detecting902additional objects12and/or regions12of interest in the already detected area(s) of the image16. For example, a neural network905detects a car12and then another neural network905detects a license plate12on the cars12. One neural network905can be responsible for detecting902objects12and/or regions12of interest in the image(s)16, and one or more additional neural network(s)905can be responsible for extracting landmarks (1005seeFIG.10) from the objects12and/or regions12of interest in the image16. For example, a neural network905detects a pothole12, and then another neural network905will identify its topmost point, bottom-most point, leftmost point, and rightmost point, and return those in a coordinate format respective to the image16, or in a coordinate format respective to the object/region12of interest.

Further, the neural network inference can be processed on the Device GPUa (in addition to the CPU111aresident in the computing infrastructure100). The neural network905can infer multiple classes12simultaneously. Further, one or more of the neural networks905can be simplified by approximating the neural network to floating-point numbers for the purpose of reducing the memory and processing requirements. Such reduced neural networks, sometimes known as Quantized neural networks, are then used on the Device101CPU111a.

As discussed, the image processing instructions905can include utilizing sensor data17to interpret/decide upon objects of interest12and/or discard data19. For example, sensor data17(such as GPS data associated with the images16) can be used (as configured in the image processing instructions905) by the device101(and/or server107a) to discard a portion/whole image frame16a,b,c,dfrom inclusion in the object data21.

Referring toFIG.10, shown are example inference results1000obtained from the image processing instructions905(e.g. neural network(s)905). Neural Network(s) Inference results1000of a processed image16, including bounding boxes1002, polygons1003, masks1004, and landmarks1005as part of the resultant processed image data20. When the system900is processing an image16through a Neural Network905, it returns inference results in the form of data20. The data20is typically a list or array of things12it is trained to find, typically known as classes12and typically represented as a class id which can be correlated with a name (i.e. class 1 is person, class 2 is car, etc) and numerical score. The score typically represents the Neural Network's905confidence in the class12as identified in the image16.

The Neural Network905can also provide additional information, per object12, as to where the object12is found in the image16in the form of a Bounding Box1002, which is typically a rectangle that is encompassing the object12. The Bounding Box1002information could be provided in a variety of formats which could be used to construct a rectangle. For example, it could be two opposing coordinates (i.e. top left, bottom right) in the rectangle, or a center coordinate provided also with width and height parameters. The Neural Network905can also provide additional information, per object12, as to where the object12is found in the image16in the form of a Polygon1003, which is typically a series of connected points that are encompassing the object12. The Polygon1003information could be provided in a variety of formats which could be used to construct it.

The Neural Network905can also provide additional information, per object12, as to key features of the object12in the form of Landmarks1005, which is typically one or more points which are expected to be present in the object12. Landmarks1005could represent many things. For example, landmarks could represent lane markings12and edge12on a road. They could represent the top and the bottom of a sign post12. They could represent the top and bottom edge of a pothole12. Neural Network(s)905can learn to identify any landmark(s)12in any object12. The coordinates for the Bounding Boxes1002, Polygons1003, and Landmarks1005may be absolute pixel coordinates in the image16(in relation to top left corner of the image), they may also be relative to another point in the image16. The coordinates may also be provided in the form of a fraction, or percent of the image16, which could then converted to the pixel representation.

The Neural Network905can also provide additional information, per object16, as to where it is found in the image16in the form of a Mask1004, which is also known as instance segmentation. The Mask1004information typically maps, on a pixel level, which pixels in the image belong to a specific object12. The system900can identify one or more object12types, or classes12, in the same image16, using one or more Neural Network(s)905. For example, it may identify signs12, potholes12, people12and cars12.

Since the system900facilitates automated incident12detection in the images16, the Neural Network(s)905can have the option automatically (e.g. in the AI process301) identify307incidents/objects12, including such as but not limited to:(a) cracks, including some or all of the following crack types: longitudinal crack(s), linear crack(s), transverse crack(s), reflection crack(s), lane joint crack(s), widening crack(s), fatigue crack(s), alligator crack(s), crocodile crack(s), block crack(s), hair crack(s), edge crack(s), edge joint crack(s), slippage crack(s), and/or shrinkage crack(s). The cracks may be further differentiated by severity, such as the extent of the road to which the cracks apply and how wide the cracks are; and/or(b) deformations and/or distortions in the pavement, including some or all of the following: rutting, shoving, corrugation, depression, and/or upheaval; and/or(c) road repairs, including some or all of the following: asphalt patching, pothole patching and/or crack sealing; and/or(d) road damage, including some or all of the following: potholes, stripping or raveling.(e) street signage issues including some or all of the following: unreflective signs, bent signs, faded signs, obstructed signs, damaged signs, crooked sign, twisted sign, and/or vandalized sign; and/or(f) manhole issues, including some or all of the following: such as raised manhole, sunken manhole, and/or open manhole; and/or(g) drainage related issues, including some or all of the following: water pooling on pavement and clogged catch basins; and/or(h) pavement marking issues, including some or all of the following: faded markings, and unreflective lane markings; and/or(i) road obstruction issues, some or all of the following: debris on road, cadavers, and construction garbage bins/dumpsters which are obstructing the right of way; and/or(j) side walk issues, including some or all of the following: trip edges, cracks, chips, distortion and/or deformation.

In addition to identifying and reporting incidents12automatically in the AI process301, the Neural Network(s)905can also infer in the process301for other purposes as well, such as but not limited to:(a) The Neural Network(s)905may assess environmental conditions which may affect the system functionality, including some or all of the following conditions: heavy rain, heavy fog, heavy snowfall, occluded windshield, daytime lighting conditions, night time lighting conditions, reflections on the windshield, and/or sun glare. The Device's101software108may then adjust its algorithm and parameters to optimize the system's performance under the identified environmental conditions. For example, an incident identified while the Vehicle's102windshield washer is in operation, the image would be discarded if the fluid spray blocked the field of view. Another example, if the road14is classified as completely covered in snow, pothole12detections would be excluded. The primary purpose of the environmental neural network905is to reduce erroneous incident occurrences, but it could also be used to report conditions and incidents. For example, if the road is covered in snow, it may be due the fact that the snow plows have not covered that road, which may be considered as an incident.(b) The Neural Network(s)905may assess the road14type in order to know which type of models, workflows, algorithms or Neural Network(s)905to apply. For example, the road14may be asphalt, concrete, gravel, dirt, trail or other types of roads.(c) The Neural Network(s)905may assess the road14conditions for the purpose of collecting road ratings for the roads which are travelled (such ratings may also be known as Pavement Condition Index). Examples of ratings may be a quantitative numerical value between 0 to 100, or a descriptive qualitative rating such as excellent rating, very good rating, good rating, fair rating, poor rating, fail rating, failed rating, failure rating, and/or other ratings descriptive of the pavement condition.(d) The Neural Network(s)905may identify object(s)12of interest representing assets which are in a good state, whereby the absence of those objects12over time may be an incident. For example, street signage (which can be damaged or dislocated by high winds), traffic signals (which can be out of order), street lights (which may be burnt), lane markings (which can fade). The presence of such objects12could be correlated with a GIS database and ruleset to generate absence incidents. If an asset has not been detected for a certain amount of time, it gets flagged in the system. For example, this could be used to indicate that a stop sign has potentially blown down due to high winds or that a speed limit sign has been obstructed by vegetation.(e) The Neural Network(s)905may identify object(s)12of interest representing object(s) which are to be redacted for privacy.

The system may, based on its programming, initiate and autonomous incident detection307process which would identify objects12or incidents of interest12and initiate the interactive image acquisition process803a.

Referring toFIGS.11,13shown are example display options for a user interface119aincluding view options such as an incident image1101, user interface1102, viewfinder1107, safety mode1103, navigation mode1104, and a background mode allowing third party apps to run1105. The device101can have a user interface119awhich may be integrated or external. For example, a smart phone or a smart camera101can have an integrated display111whereas an embedded computer101may or may not have it, depending on its configuration. The device's101software108, depending on the programming, may display different options.

The device101may have a user interface1102screen that will allow the vehicle's102aoperator to choose different options pertaining to the operation of the system10,10′. The user interface and its settings, options and/or menus can be accessible through a touchscreen, built in button and/or a remote control. Typically, when the device101is initially installed or operated in a vehicle102a, a viewfinder1101option will be enabled, which will allow the vehicle102aoperator to know that the device's101cameras500are aligned properly. The display111of the device101can be configured to minimize the distraction to the driver by turning off the screen or by displaying a warning message to not operate the device101, when the vehicle102ais in motion or even by disabling the user interface to prevent driver from interacting with the device when the vehicle102ais travelling faster that a configurable threshold.

The display111can also be configured to assist the driver during operation by displaying a navigation1104screen. The navigation screen1104may show the device's101current position, and/or highlighted routes of roads14to patrol via a map interface1104. The device's101software108may also run as a service in the background of the device101, allowing a third party application1105to run on the device101while the software108is running in the background. For example, once the software108is started, other applications related to navigation, automated vehicle102alocation, work order management, dashcam video recorders, and other applications could be launched in the foreground. The third party applications may be launched by the software108and/or by the device's101operator.

Referring toFIG.12, shown is an example incident/object12identification and reporting operation1200. Depicted by example is Vehicle102aequipped with a Device101. The Device101identifies object(s)12of interest which are to be reported1201. The Device101then transmits data1202to the Server(s)107a. The Server(s)107ahave files1203, one or more database(s)1204, and Software(s)1205. The Server(s)107acommunicate with Client(s)1208(which may be the devices101or separate computing devices with a browser or software that provides access to the system1200) which allow users to handle the incidents12using a user interface1209which may present the incidents12in a variety of ways including a map view1210, list view1211, and gallery view1212. Object(s)12of interest identified and reported1201pertains to possible inference results (e.g. resultant processed object data20,20′ and associated sensor data17) as described in further detail inFIG.10—Neural Network(s) Inference Results1000.

Further, for example, data transmitted1202illustrates what typical incident data12contains, which can include some or all of the following:(a) IDs, which may include the Device101ID, image ID, detection ID, user ID, and/or other ID's allowing to associate the data;(b) Longitude: east-west position of a point on the Earth's surface. Different countries may use different names or system to describe the point;(c) Latitude: north-south position of a point on the Earth's surface. Different countries may use different names or system to describe the point;(d) Sensor(s) Data17: described in greater detail with respect toFIG.7—Sensors700;(e) Detected Object(s) Data: described in greater detail in Neural Network(s) Inference Results1000;(f) Date & Time: a date and time signature pertaining to the time and date that the incident was identified;(g) Image: a picture of the incident capture by the Device's101camera(s)500. The image may be redacted; and/or(h) Other data: Other data pertains to other data that may be derived by the Device's101software108from its configuration, settings, sensor(s), database(s). For example, the direction to which the Vehicle102ais travelling may be determined based on the Device's101sensors700. Other examples of other data may include the road segment on which the incident was obtained, or the direction that the Vehicle102awas facing, or a geo-zone in which the incident was obtained.

In view of the above, the incident data21is transmitted to the Server(s)107awhere it is processed by the Server(s) software(s)1205and organized and stored in database(s)1204. Some data, such as uploaded incident images20,16containing the incidents12, may be stored in the form of Files1203. The server107aalso provides for client(s)1208to securely log in to access a user interface1209, which may be either a web application that can be accessed using a web browser or a client/server application that uses physical installation to a computer or a smartphone101. Through the user interface1209, clients can view incidents12detected by the device101, which have already been uploaded to the server107a, which can be visualized in a variety of ways, such as but not limited to:(a) a Map View1210which depicts a map, along with pins representing detected incidents and known assets;(b) a List View1211which depicts a table along with fields related to each incident; and/or(c) a Gallery View1212which depicts a gallery of incidents in the form of clickable images.

Further, by example, users can click on the pins to display more information1214about the incident12such as image, details of the detection, severity, the date and time the detection occurred. Through a click of a button, the detected incidents12can also be presented in a gallery view layout, where a thumbnail image of each incidents can be presented in a grid.

Client(s)1208pertains to software that is used to access the system10. It can typically be a web browser, but it may also be a dedicated desktop application and/or a smartphone app. The Client(s)1208user interface1209may have different views to present the data the user. For example, when selecting the incident12or asset for which it is related in the appropriate view, more information is displayed to the user.

The Server(s)107may process the information (object data21) in a variety of ways, for example, the Server(s)107can also associate incidents12, through their GPS coordinates, to a road14network segment, which is a representation of a segment of a road14, which typically includes geospatial and descriptive data, such as points, features and other fields—for example the class of the road (highway, local, regional), the street name, and/or the address range which it covers. Road segments are typically extracted from a geospatial database such as a shape file, KML file, KMZ file, XML, GML and/or other such popular formats used for the modelling, transport and storage of geographic information.

The Server(s)107may also have an asset database, particularly for road signs, manholes, and catch basins, where the GPS coordinates, direction of travel and type of asset are logged in for every detected asset when they are detected on the Device101. This database can be used for inventory purposes or as a list for manual inspections.

The GPS coordinates17and sensor information17of the incident12or asset may further processed on the server to determine additional insights, including some or all of the following:1) Nearest address;2) Nearest road intersection; and/or3) Nearest major road intersection.

It is recognised that the server107acan be a physical server connected to the internet18. Alternatively, the server107ais a virtual server connected to the internet18, and whereas it may be hosted on one physical machine or on a server cluster. Alternatively, the server107ais cloud based and connected to the internet18.

Referring toFIGS.1,13, a further embodiment of the system10,10′ is shown that is designed to remotely acquire images16,117(hereafter referred to as images117) and related data17,20,118,119(hereafter referred to as data118,119) of road related incidents12(seeFIG.1and related described examples) from a device101intended for use in a moving vehicle102autilizing software108(including artificial intelligence modelling905—seeFIG.9). The AI device101can utilize image processing906functions of the software108, including using machine learning models (further described by example below) to automatically detect objects of interest within an images117, as well as perform other inference operations such as but not limited to, classification, image segmentation, or other actions related to image based artificial intelligence. Certain embodiments of the present invention can relate to controlling a series of processes in the AI device101, with respect to acquiring of the images117, identification of objects(s) of interest, and/or the acquiring of the sensor data118.

Further, in some embodiments, a trigger module112can be used remotely control device101operation using a set of commands (e.g. trigger signal124) exchanged between the AI device101and the trigger module112. In some embodiments, voice commands116in the form of vocalized words or phrases can control the device101operation. In any event, it is recognized that the use of the trigger module112(for trigger signals124) and/or voice commands116can be used to augment the operation of the software108used to automatically detect the object(s)12of interest in the acquired images117, i.e. utilizing the interactive incident acquisition process803a(seeFIG.15)

As further described below, with relation to signals124,116, the AI process301uses the AI algorithms905to process the images117as acquired by the camera107. The interactive interactive acquisition process803acan be used by the operator to manually input or otherwise augment the incident data120autonomously detected307by the camera107and associated AI processing905.

The system can utilize the device101, designed to automatically collect images117from the device's camera107, additional data118from the device's sensors105, as well as additional data from the device's101operator (i.e. augmentation of the software operation108) using the external trigger signal124and/or voice command/signal116. The device101processes the incident data120(including the data117,118,119for example), in a workflow (see Examples inFIG.9,15,16) for the purposes of conducting inference on the data120, using artificial intelligence algorithms905of the software108. The artificial intelligence algorithms/methods905, in the form of software algorithms, can be deployed directly on the device101and utilize the device's101hardware (e.g. Volatile Memory103, Non-Volatile Memory104, CPU106, GPU110) in order to perform inference on images117. Any or all of the software108pertaining to image117processing can also be deployed on remote server(s)123, where the system10,10′ can utilize cloud computing infrastructure in order to perform analysis on the acquired incident data120.

Accordingly, the device101and associated software108performs the automated acquisition process307in order to generate the incident data120. However, the operator can augment the content of the incident data120, via implementing the interactive acquisition process803ausing the trigger signals124and/or voice command116signals, in order to provide supplemental incident data117,118,119as part of the incident data120, as further described below (seeFIG.15).

Referring toFIGS.1and13, the device101is affixed to the vehicle102atravelling along the road14. When in operation, the device101is typically configured to collect visual information of a scene in the form of digital images117, and processes the image data117using the appropriate algorithms and methods905typically associated with computer vision tasks. Examples can include some or all of the following, in a workflow or independently: cropping, resizing or rescaling, transformation, image classification, object detection, semantic and instance segmentation, scene reconstruction. The device101can also process associated data such as data118acquired from other sensors105or from the operator via voice commands116or a trigger signals124in its software108workflows, via using the interactive acquisition process803a.

The components that make up the device101are typically inclusive of a central processing unit (CPU)106and/or graphics processing unit (GPU)110, memory including high speed volatile memory103and low speed non-volatile memory104. All of which facilitate the device's101execution of the software108instructions (e.g. AI algorithms905). Examples of software108types that could be present on the device101include operating system(s) software, drivers for the various hardware interfaces, software applications, databases, and software modules which may include image processing instructions, AI libraries, text to speech software. The device's non-volatile memory104can be used for storing these software108instructions necessary for operation, along with storing associated files necessary for operation. Such files may include operating system files, component driver files, application files, database files, configuration files, log files, media (such as images, audio, and video clips), and other files. The device's101operating system can include Windows operating system, Android operating system, Linux operating system, or other operating systems. The above are some example of types of software108and software108files which are associated with different embodiments of the device101, but other types of software108and software108files may also be present.

In one embodiment, the device101can be packaged in a housing such that it contains all of the components of the device101, whereas in other embodiments some components may reside outside of the housing and connected via common interface connector or using a wireless connection. The device101can be suited for mobile operation in a vehicle102aand be installed in a manner that facilitates ease of transfer between separate vehicles102a. Electronic devices101that may be suitable for use can include currently of the shelf smart phone devices, smart camera, or embedded computer systems. Smartphone101devices can include currently available phones including Samsung Galaxy S-series phones, Samsung Galaxy A-series phones, Samsung Note phones, LG G-series phones, iPhone 11 models, and can include the majority of modern models of smartphones, and future models thereof. Further, the device101may be a smart-camera101capable of executing instructions related to machine learning algorithms, including artificial intelligence neural networks905. Devices101can include those containing chipsets supportive of processing highly parallel operations, such as devices containing large graphics processing units (GPU), and/or neural processing units (NPU), and/or Tensor Processing Units (TPU). Examples of chipsets include, Intel Movidius, Nvidia CUDA, Texas Instruments Sitara, Qualcomm Adreno series, etc.

The device101can be equipped with one or more camera(s)107, and/or different types of cameras107to acquire incident images117while travelling along a road surface. The camera107or cameras107can be contained internally or externally, in relation to the device101. In the case of external camera(s)107, the device can interface with the camera107by use of a wired or wireless interface. Examples of types of cameras107include, telephoto, wide angle, infrared, thermal cameras107, or camera107able to otherwise operate at different focal lengths, resolutions, shutter speeds and light spectrums. Different camera imagers500,107(seeFIG.5) can be used for different use cases. Further, the device101can utilize multiple cameras107simultaneously, including one or more contained internally or externally. In some cases, different cameras107can be used under different circumstances. For example, at night when lighting conditions are poor, the device101can switch cameras107to one that is optimized for night vision. This can be done by the use of software108controls, based on a schedule (e.g. sunset timer), image parameters (e.g. exposure and/or brightness), or a sensor105(e.g. light sensor). Camera107may also be switched manually by a system user selecting a certain camera107mode based on the desired setting.

The device101can include a display111for displaying relevant visual information to the user, which may be integrated into the device101or provided externally. Additionally, the display111interface may be coupled with a user interface119aallowing users to configure different options or settings pertaining to the operation of the system10,10′. The user interface119aand its settings, options and/or menus can be accessible through a touch screen, built in button or button(s), gestures, and/or remote control. In initial installation of the device101in a vehicle102a, typically a viewfinder option may be enabled, allowing the user to check alignment of the device's101camera107with respect to the road14. The device101display111may also show notifications about incidents which are manually tagged by the device101operator, during operation of the software108(and associated AI algorithms905) via the manual acquisition process803a, as the vehicle102ais travelling along the roadway14.

The device101can include audio interface(s)102for acquiring and processing audio data116(voice commands) pertinent to the function of some embodiments of the current system10,10′. Audio processing is used for utilizing voice trigger capabilities for voice commands116in order to trigger a response within the software108to manually initiate a triggered incident data120acquisition process803a. A microphone125, coupled to the audio interface102can also be used for functionality of the process. Depending on the embodiment, a pre-amplification circuit (not shown) can be integrated into the device audio interface102, or into the microphone125. The microphone125can be embedded in the device101or connected externally. The device101may also include an internal or external speaker126. The speaker126can be coupled to the audio interface102. The audio amplifier can be integrated into the device101, the speaker126, or an enclosure including the speaker126. The speaker126could be used to provide audible confirmations to the device101operator, to indicate whether an incident was captured successfully, as directed by the operator using the triggered operation (e.g. interactive incident acquisition process803a). Audible confirmations can be a pre-recorded sound, such as a camera sound or a recorded sound of a person. It may also be a text to speech indication, in which the device101would be speaking to the operator.

The device101can store acquired incident images117, sensor data118, and operator metadata119into a storage medium104prior to transmitting to the server123. It can do so to optimize power or data usage, or simply to queue data in the case that a reliable network121connection isn't available. Appropriate storage mediums for the task can include various forms of non-volatile memory104. The acquired data120can be stored temporarily in the non-volatile memory104, and deleted once successfully uploaded to the server123. Furthermore, storage functionalities can include storing files associated with operating system(s), component drivers, application(s), media, and a plurality of other files associated with software applications. Acquisition images117stored on the non-volatile memory can be stored in various file formats, including JPEG, Bitmap, WebP, PNG, and other common image formats. Data118,119acquired from the various sensors can also be stored in various file formats, including CSV, XML, JSON, in software databases or other formats for storing data120. The non-volatile memory104can be embedded within the device's101housing, or in the form of add-on storage that is connected to the device101or inserted into storage slot(s) provided by the device101. Examples of different types of non-volatile storages104currently popular with embedded systems include hard drives (whether disc based or solid state memory), and memory cards (such as microSD). Other types of non-volatile storage104may be used if the device101interfaces support it.

The device101can include a mounting component103a, which couples the device to the body of a vehicle102a, such that the imager107of the device101has a viewpoint of the road surface14and any desired adjacent surroundings13. Typically, the device101has the option to be mounted on the vehicle's102awindshield, dashboard, side windows, back window, roof, or frame. The device101may be mounted in the vehicle102aor external to it. The mounting component103acan have different configurations intended for use in different vehicle102atypes or even different device101configurations. For example, utilizing different a different mounting component103aif the device101is a smartphone, smart camera, or embedded computer equipped with external camera(s)107.

The artificial intelligence device101typically includes software108,905configured to perform tasks useful in computer vision inferencing, but also for other artificial intelligence processing, such as sensor data analysis and audio transcription. The software108,905can utilize various algorithms and models suitable for these types of tasks, by configuring the software108,905to execute the necessary algorithms in certain sequential processes, and/or parallel processes. The device's101memory99,104can be loaded with algorithms, and models, where the software108,905is configured to run machine learning models, including artificial neural networks or other types of inferencing methods utilizing artificial intelligence905. The device101may include common wired and wireless interfaces which can facilitate it to connect to a plurality of external components.

Referring toFIG.13, the trigger module112can be packaged in a housing such that it contains all of the components of the trigger module112. The components comprising the trigger module112can include a microcontroller113and/or integrated circuits113, used for monitoring the trigger module's inputs114. If an input114(e.g. touchscreen, dedicated buttons, etc.) has been activated, then the microcontroller113executes a set of instructions to send a trigger signal124through the trigger's communication interface115to the AI device101that initiates the interactive incident image/data process803a.

It is recognized that incident reporting can take place by use of voice commands116. In order to trigger the device101to capture incident images117and related sensor data118, the physical triggering mechanism112could be replaced altogether or otherwise substituted on a case by case basis, by triggering the process803ausing predefined speech commands116. The AI device101can have an audio interface102with audio input and output capabilities. The audio interface102can comprise an analog to digital converter used for processing analog voltage signals, from a microphone125, into digital data, where it can be further processed. It can also comprise a digital to audio converter used for processing digital data into analog voltage signals, where it can be processed by a speaker126.

The device101can be equipped with a microphone125for processing audio inputs116. The microphone125can be internal or external in relation to the device101, and can be comprised of different microphone125technologies. For example, it can be comprised of microphone125technologies such as dynamic, electret-condenser, MEMS, or other types of microphones125used to process audio into analog signals. The speaker126can be internal or external in relation to the device, and can be used to transmit audio signals sent to it by the device101. The speaker126can be comprised of various speaker126technologies such as, an electro-dynamic speaker, piezoelectric speaker, MEMS speaker, or other types of speakers used to transmit analog audio signals. The speaker126and microphone125may have additional circuits or converting digital to analog or analog to digital signals (not shown).

Audio commands116can come in the form of predefined human speech commands116that is captured by the microphone125and processed through software108algorithms on the device101. For example, a predefined speech command116can include the expression “Incident1” that will cause the device101to perform the same action as activating a dedicated “Incident1” input114via the trigger module112. This voice/trigger activated action (of the device101) could include some or all of the following steps: capturing the incident image117, and relevant sensor data118, tagging119the image117with “Incident1”, processing the image through the AI model109, storing the incident data120, and then sending incident data120to the server123. The manual incident tagging119via voice commands116or signal124may be a separate process that the automated one all together (meaning no AI image operations905are done), or used in conjunction, before or after the AI905process such that the induced action (of the device101as caused by trigger116,124) is used to augment the incident data120collected via the autonomous process307. The same process803acould also be executed using the command116“road damage”, or “road hazard”, or any other collection of words in human language. The actual command116used for triggering the process803acan be the same as the word used to tag119the image117, or the words can be different. For example, the predefined command “Alpha” can execute the process803a, but the image117can be tagged119with “stop sign damage”. In this way, the operator can utilize the trigger signal(s)116,124to initiate manual incidents capture803a, or alternatively, augment, via the process803a, the automated process307.

Multiple speech commands116can be used to trigger the above process803a, as well as the predefined command116words can be configurable by the user. The user of the system10,10′ can choose from a list of words in their language of choice, but the amount chosen to be used as commands116can be limited as to not have the device101trigger the process803abased on every word it picks up. The chosen words/phrases to be used as trigger commands124,116can correlate or not correlate with the tags chosen for each input on the physical trigger module112. For example, the same tags119assigned to each input114on the trigger module112can correspond to the predefined commands116chosen to trigger the process803aand tag images119. Alternatively, the tags119assigned to each input114on the trigger module112can be different from the predefined commands116.

The voice command116functionality can be configured to be used in conjunction with the physical trigger module112. For example, the input114on the trigger module112can be activated to initiate the incident image capturing process803a, while the incident tag119is taken from the voice input116. The voice input116can be processed by the speech recognition software108in order to extract the exact word or phrase, with the extracted word/phrase tagged119with the incident data120. The words/phrases can be predetermined, or not predetermined and include any word/phrase in the user's language, since in this embodiment, the process803amay be initiated when the physical input114on the trigger module112is activated.

The device's101audio interface102and software108can be configured to constantly process incoming audio116from the microphone125while the system is in operation. The devices101software108can utilize speech recognition algorithms in order to recognize whether a command116, from the list of predefined commands116, has been vocalized. An activation command116may also be setup, whereas the activation command116is an uncommon word or phrase, which would then notify the device's101software108to expect a follow up command116. For example, saying “Hey ROVER, [activation command] tag shoulder drop off [incident command]””. The activation command116may be processed together with the incident command116, or separately. If a command116, in the form of an audio signal, is recognized by the software108algorithms used to recognize speech, then the incident image capturing process803awill be initiated. If no command116is recognized, then the device101continues its regular operation301. The choice of speech recognition algorithms can be a variety of models, methods and algorithms or natural language processing applications. It can include statistical methods, audio processing functions, machine learning, deep learning, and neural networks.

Alternatively, speech recognition can be performed through the remote server123, subsequent to the device101transmitting audio command data116to remote server123over the chosen network121. Speech recognition algorithms deployed on the server123can be configured to perform inference on the incoming audio command data116and return a response with the recognized word or phrase, which would then be incorporated as part of the workflow of the process803a. The device101can further use the returned statement to make a decision on whether the command116is valid and initiate the process or whether to ignore the command116.

In another embodiment, a hybrid function could be used, whereas the device101would recognize an activation command116(for example, “hey ROVER”) using the device's software108, after which it will record an audio clip119of the user command116to be sent to a remote server123for processing, and get in response a word or a phrase116, which would be used to tag119the image (if the result is valid).

In another embodiment, it could also simply store the audio clip119with the incident data120to be transmitted over the network121, and be processed and stored or discarded directly on the server123.

The device101can include a network interface122, used to communicate with and transfer data120to the server123over a network121. The network interface122may be contained internally or externally, and can interface with the device101via wired connection or wireless connection. It can be used to communicate with the server123over a cellular network121, wireless LAN121, and/or wireless WAN121. In the case of a cellular network121connection, the interface can utilize technologies such as 3G, 4G, LTE, 5G, or other technologies used to access cellular towers. Further, it can operate at common frequencies such as 2.4 ghz, 5 ghz, or other common frequencies associated with IEEE 802.11 or other wireless standards. The network connection provides for communication with the server123on a constant, frequent or periodic basis, allowing acquired incident data120to be transmitted as network communications take place. The device's101network interface122can utilize cellular networks121in scenarios where cellular network121connections are readily available, when cellular data costs are not prohibitive, when faster incident uploads are required, and/or when the device101has no access to other wireless networks.

Furthermore, the device101can upload acquired incident data120intermittently when a connection is not accessible. For example, the device101can include non-volatile storage104that can store the images117, sensor data118and incident metadata119temporarily and when a connection is available, send the data120through the network121to the server123. Some users may choose to only use wireless LAN connectivity in order to forgo cellular networks to save on data120costs. In this case, acquired incident data120could be stored until the device101has access to a wireless access point connected to the internet. The device101can be configured to transmit data120at a scheduled rate, in order to benefit from improved performance, power consumption, and/or heat management.

Additionally, images117, sensor data118, and metadata119may be processed to be more optimized for network transmission, which may include format conversion, compression, and/or encryption. Data120may be temporarily stored on the device101in the event that a network connection121is unavailable. In the event that a network connection121is available, transmitting the data120for further processing, storage, access and control. In some embodiments, acquired images117can be further processed where the image processing instructions905are configured to automatically detect and redact objects in the images117pertaining to personally identifiable information, e.g. people, cars, and/or license plates12. These image processing instructions can be performed directly on the device101, utilizing its AI functionality905, or performed on the server123once the images117have been uploaded.

Referring toFIGS.7,13, the device101can be equipped with a variety of sensors105,700which may be contained externally or internally in relation to the device101. The types of sensors105can be comprised of motion sensors, position sensors, environment sensors, location sensor, and other sensors commonly utilized in smartphone devices or embedded camera systems. The location sensor105can be used to obtain location data118which can be used to determine precise coordinates in regards to where incidents occurred. Location sensors105provide location based information using satellites such as GNSS, GPS, Glonass, Galileo, and in some cases cellular towers in order to determine positioning data of the device101. The sensor data118is usually processed by the software108prior to being sent to the server123. Further, the sensor data118along with the acquired images117can be stored on the device's101non-volatile memory104in the form of file or database entries prior to transmission to the server123. In addition, the device101may contain additional sensors105, which provide data118that can be used to determine device acceleration, orientation, magnetic pole, velocity, shock, vibration and other data118related to position and motion. An accelerometer105would be an example of a motion sensor105. It can be used to measure acceleration applied in the x-y-z axes. A gyroscope105is another example of a motion sensor that may be utilized, where the gyroscope105can be used to measure the rate of rotation about the x-y-z axes. A rotational vector sensor105is an example of a position sensor that can be used to measure a wide range of motion-related tasks, such as detecting gestures, monitoring angular change, and monitoring relative orientation changes. The device101may also include a magnetometer105as an additional position sensor. It can be used to determine the device's101position in relation the magnetic north. Other sensors105can include sensors105such as environment sensors105that typically monitor environmental properties, such as ambient temperature, device temperature, light and humidity. Such sensors105can also be useful for monitoring the hardware and/or operating environment of the device101.

Any and all of the aforementioned sensor data118can then be associated with images117acquired by the device's camera(s)107, in order to obtain additional insights. In many cases the acquired data118,119will need further processing to obtain the necessary information about an incident12. For example, geo-positioning of an incident of interest12based on where the image117was acquired, vehicle speed, and/or zoning location. The sensors105and camera(s)107provide data118,119to be initially processed by the software108, then transmitted to the server123or stored temporarily on the device's non-volatile memory104if a network connection121is unavailable at the time. The data118,119may be stored as file(s)123in variety of formats such as xml, csv, txt, or in a proprietary format. The data118may be stored in a database, as well as stored and transmitted in encrypted or non-encrypted format. The incident data118,119and images117may be further processed on the server123using additional processing instructions deployed on the server123including, correlation with road segments, inspected roads and/or assets, and other useful information. An accelerometer (motion sensor), magnetometer (position sensor), and location sensors could be used by the software108to determine the vehicle's102adirection of travel, which can be used to determine which side of the road14a vehicle102ais travelling on when an incident12is acquired. It can also assist in determining more precise incident location in relation to the device101or vehicle102a.

The system10,10′ is intended to be used in a moving vehicle102aoperated by the user. The device101is typically mounted to the structure of a vehicle102ausing a mounting component103a, such that the device's camera107has a viewpoint of the road surface14including adjacent surrounding13(e.g. roadside, sidewalk, overhead). During operation, the device101performs inference on an incoming stream of images117obtained from the connected camera107. Based on results of the inference operations (typically performed using image processing and AI/machine learning models905) of the incoming images117, the device's101preconfigured software108determines whether to discard an image117or keep it, where it can be uploaded to a server123for further storage, access and control. In addition, the device's101software108may also utilize sensor data118, trigger data124, or voice command116data in order to determine whether to discard an image117, or bundle it into incident data120for storage and transmission to the server123.

In many cases, the vehicle102acan be operated on behalf of an organization which can be governmental, quasi-governmental or a private company. It can also be used voluntarily by individuals as a crowd-sourced application. Examples of governmental organization include all levels of government, including national, federal or republic governments; provincial, territorial or state government; municipal government, including municipalities, upper tier local governments, or lower tier local governments. The governmental organization may also be a special organization, such as a reserve, resort, or other names that are used to describe local governments of a certain geography or population. Examples of quasi-governmental organization would be government-owned or supported organizations. Those could be organizations established as part of a public-private partnership or a concession to build, maintain and/or operate an asset or a service over a period of time. They could be separately incorporated but the government may have full ownership, majority ownership, or minority ownership. Example of quasi-governmental organizations include toll road concession companies, bridge concession companies, transportation and/or transit authorities, and/or utility or telecom companies. A private company can simply be a private company that is the owner of the asset that is to be inspected, or contracted on behalf of the owner to do so. The vehicle can be a service vehicle dedicated to patrolling an area for the specific purpose of identifying incidents/objects on behalf of the organization. The vehicle102acan be a car, truck, van, golf cart, tractor, ATV, bicycle, e-bike, motorbike/motorcycle, snowmobile, or customized utility vehicle. For example, the vehicle's102aprimary purpose can be something other than acquiring incidents12.

It is recognized that the software108is an integral part of the system, implementing a plurality of instructions necessary for operation of the system. The software108can facilitate various instructions that make operation of the system more accessible users. For example, the software can facilitate for field of view adjustments. The field of view adjustment can be optical zoom adjustment, if supported by the peripheral camera107of an embedded computer system. If further magnification is required to calibrate the camera's107field of view, the digital zoom level can be adjusted through the software108to achieve the desired optimal field of view. The software108may facilitate selecting from a variety of internal camera(s)107or external camera(s)107in order to adjust the field of view. Different fields of view may be optimal for different use cases on vehicles102a. As well as different fields of view dependent on the type of incident or object12being acquired, for example signs may require wider field view, opposed to road defects12requiring narrow field views. The software108may utilize different trigger inputs114or voice commands116for different camera(s)107.

Multiple processing steps in the form of software instructions108can take place for the purposes of cleaning, filtering, cropping, adjusting, scaling and/or other instructions used for processing images117, sensor data118operations or metadata119generation. Acquired images117, sensor data118, and metadata119are typically processed by the device101prior to resultant data120being transmitted121to the server123. Where the server123may implement further processing instructions on the incoming data120, prior to subsequent storage on the server123. It is recognized, that the software instructions108can be an embodiment of the device101, the server123, or a combination of both server123and device101. The device101and server123can have different variations, configurations, and/or parameters relating to the operations taken.

The software108can include image operations and AI instructions905for image processing and AI inference, as well as workflows for the image117processing capabilities and neural network(s)905inference operations. Some examples of inference operations include object detection, image classification, and instance segmentation. The AI and Image processing operations905can further analyze images117, whether acquired under the trigger112or voice command116or regular device101operation to redact or discard certain image117data. The workflows may include temporary storage of the image117on the device's non-volatile memory104, whether redacted or unchanged, prior to transmission to the server123. It is recognized that the workflows can include a plurality of different combinations or number of operations in any order, as configured, in order to image process, as well as use AI to identify, classify, and segment any objects in the images117under consideration.

The software108may facilitate cropping portions of an image117. This can be used for extracting parts of an incident image117that can contain personally identifiable information, extracted by additional image processing and/or inference activities. For example, a car12can be detected by a neural network905and then cropped from the image117. The cropped car12may then be either redacted or processed through a neural network905that is trained to identify license plates12in a picture. Operations related to other forms of redaction such as object12redaction and/or image redaction are also considered image processing operations as performed by the software108and related instructions.

Referring toFIG.9, in the case of images117, the variety of image processing and AI neural network(s)905operations may be organized in workflows. Some embodiments of sample workflows that may be configured are workflows which may include neural network(s)905that may be utilized as a detector that typically identifies an object of interest12in image(s)117and location of the object12, in the form of a bounding box. The detector may also provide a score, typically known as confidence. The neural network(s)905can also be utilized as a classifier. A classifier has a list of potential classes, or object types, in which it is trained to identify in a picture. A classifier typically returns a list of potential objects in the image117sorted by the models confidence. The neural network(s)905can be utilized as a segmentor. A segmentor typically segments image(s)117into regions. The regions are then typically predicted to belong to a certain class, or type. The neural network(s)905can be designed and/or optimized to be used on the device's101GPU110, CPU106or both. The workflows may utilize one or more neural network(s)905and the networks may be used in sequence. One neural network905can be responsible for detecting objects12and/or regions of interest in the image117, and an additional neural network905responsible for classifying detected objects12. One neural network905can be responsible for detecting objects12and or regions of interest in the image117, and one or more additional neural network(s)905is responsible for detecting additional objects12and/or regions of interest in the already detected area of the image117. For example, one neural network905detects a car12and then another neural network905detects license plates12on the cars12. One neural network905can be responsible for detecting objects12and/or regions of interest in the image117, and additional networks905responsible for extracting landmarks from objects12and/or regions of interest.

Referring toFIGS.13,14, the server123is responsible for the organization, storage, processing and disseminating of the incident images117and other data118119uploaded by the device101. A single server123can host a plurality of users, whether governmental users, quasi-governmental users, or private organizations. A single server123can communicate with a plurality of devices101as clients of the server123. The incident data120may be secured by encryption. It may also be organized into files, or database entries, and organized in a way that prevents other users from accessing data120that is not theirs, unless otherwise specified. As the server123receives incident data120from a device101via the internet through a network121. The images117containing incidents of interest12are typically stored in one or more folders contained within the server123. In some embodiments of the invention, these folders can be allocated according to the particular device101(such as unique ID or name) used for capture, the user's organization (such as unique id or name), and time at which they are acquired. The structure could use descriptive text or a hashed derivative.

Furthermore, the incident data120can be further segmented by preconfigured incident12types in the form of tags/labels130, such as incident1,2,3and so on, or pothole, road hazard, obstructed sign and so on. These incident12tags/labels130correspond to the incidents12triggered by the trigger module112. For example, incident trigger1on the trigger module112can be configured to be labeled as road damage130, or obstruction130, or vandalism130, or any other label130the user decides to use. The incidents tags or labels130can also be based on voice commands116. The image data117can be further organized by date, road, municipality, or other properties. The server123can further process the images117to remove, transform or redact portions of data120. The discard data19′ can include personally identifiable information that may be included in an image117. Sensor data118and derivative data119, such as geographical coordinates, direction of travel, date/time, pitch and other obtainable data118119can be associated to each incident images117as well as stored into one or more database(s)202on the server123.

While the invention may be deployed on a physical server123, the invention's server123functions may be segmented by various factors. It may be segmented by geography, whereby the system may have separate servers for Canada, USA, or other countries. It may be segmented by architectural function, for example DNS, runtime, database, storage, image processing, or other functions. It may be segmented by capacity, for example 1-1010 users on one server123, 1011-2000 on another server123. It may be segmented logically, such as a having virtual machines that run on server123cluster, or cloud. Many cloud providers, such as Amazon AWS, Google Cloud, Microsoft Azure, and other cloud providers offer a variety of ways to spin-up instances of servers123on a dedicated basis or on demand to customize the actual implementation of the server123infrastructure. Therefore, the word server123and servers123will be used interchangeably throughout the description, figures, and claims, as the system could be setup to use one or more physical, virtual or cloud based server123. In some instances, the server(s)123may be hosted on premises using dedicated infrastructure provided by a customer.

In terms of format, a digital image117can be an image containing digital data content representing picture elements, also known as pixels, each with finite, discrete quantities of numeric representation for its color intensity or grey level that is an output from its two-dimensional functions fed as input by its spatial coordinates denoted with x, y. The image would be acquired either as raw image data available in various formats such as YUV, RGB, HSL, HSV, or other image color spaces and encodings available from the device's101camera107. It is recognized that the images117can be compressed using any known or available compression technology, before incident data120is sent to the server123.

Additionally, typical data120transmitted to the server123may contain some or all of the following:(a) IDs, which may include the device ID, image ID, incident ID, user ID, and/or other ID's allowing to associate uniquely associate different data elements.(b) Longitude: east-west position of a point on the Earth's surface. different countries may use different names or systems;(c) Latitude: north-south position of a point on the Earth's surface. different countries may use different names or systems;(d) Sensor data118(e) Date & time: a date and time signature pertaining to the time and date that an incident data120was acquired;(f) Image117: a picture of the incident capture by the Device's101camera(s)107;(g) Other data119: pertains to other data that may be derived by the devices101software108from its configuration, AI capabilities905, sensors105or user inputs such as trigger112or voice commands116.

In view of the above, incident data120is transmitted to the server (s)123where it is processed by the server(s)123software(s)203and organized and stored in database(s)202. Some data, such as uploaded incident images117may be stored in the form of files204. The server123also provides for client(s)201to securely log in to access a user interface, which may be either a web application that can be accessed using a web browser, or application that uses physical installation on a computer or smartphone. Through the user interface, clients201can view incidents transmitted by the device101, which have already been uploaded to the server123, which can be visualized in a variety of ways, such as but not limited to:(a) A map view which depicts a map, along with pins representing acquired incidents12;(b) A list view which depicts a table along with fields related to each incident12; and/or(c) A gallery view which depicts a gallery of incidents12in the form of clickable images117.

Further, by example, users can click on pins to display more information about the incident12such as image, details of the incident, date and time incident was obtained, and other incident data120. Through a click of a button, the acquired incidents12can also be presented in a gallery view layout, where a thumbnail image of each incident is presented in a grid.

Client(s)201pertain to software that is used to access the system10,10′. It can typically be a web browser, but it may also be a dedicated desktop application and/or smartphone application. The client(s)201user interface may have different views to present the incident data120to the user. For example, when selecting the incident12for which it is related in the appropriate view, more information is displayed to the user.

The server123(s) may process the information in a variety of ways, for example, the server123(s) can associate incidents12, through their GPS coordinates, to a road network segment, which is representative of a segment of road, which typically includes geospatial and descriptive data, such as points, features and other fields. For example, the class of the road (highway, local, regional), the street name, and/or the address range which it covers. Road segments are typically extracted from a geospatial database such as a shape file, KML file, KMZ file, XML, GML and/or other such popular formats used the modelling, transport and storage of geographic information.

The server123(s) may also have an asset database, which may describe a variety of assets, such as road signs, manholes, and catch basins. The server123may automatically suggest a nearby asset for a certain incident12type (i.e. raised manhole) identified by a user. The association of incident to potential assets may be done on the device101, the server123or both.

It is recognized that the server123can be a physical server123connected to the internet121. Alternatively, the server123can be a virtual server123connected to the internet121, where it may be hosted on one physical machine or on a server123cluster. Or, the server (s)123may be cloud based and connected to the internet121.

Referring toFIG.15, shown is the utilization of the interactive process803aworking in conjunction to the autonomous process307. This embodiment describes a state diagram displaying the various modules used to provide further functionality to devices101with AI functionality905and their interaction. Remotely controllable operations consist of initiating an incident acquisition process803athat includes capturing incident data120including digital images117of the road related incidents12, as directed by the operator via triggers116,124. The incident acquisition process803acan be initiated by the trigger activation124or voice command116. In the event of a voice command116, this would be received and analyzed by a speech to text module302, which helps to determine whether to capture the incident12and the incident information120via the process803a. In the event of an unrecognized voice command306, the system defaults to its regular AI functionality301, as discussed herein. In such case, depending on different embodiments, no incident data120can be captured, or incident data120can be discarded from storage304in memory103or on local storage104. In the event of a recognized voice command305, the system10,10′ moves to the acquisition state803a, which captures one or more of the following: the incident image117; sensor data118; and incident metadata119, which are then packaged to form incident data120suitable for storage304and transmission305to the server123.

The storage state304invokes a process that stores incident data120that has been obtained from the acquisition process803a. The storage state304can store the incident data120in a volatile memory103buffer or on a non-volatile memory104in order to store incident data120. The default AI functionality301is typically configured to obtain image data based on a separate set of rules from the incident acquisition process803a, and does so on an autonomous basis. The AI process301can also invoke the incident acquisition process803ain which it can utilize for storing images117, sensor data118and other metadata119derived from its image analysis and programming, and in the event that it considers the image117to be a positive match based on its configuration and programming, which also leads to storage304and transmission305.

The transmission state305is invoked automatically when the incident data120is ready for transmission. Typically, all data stored on the device's memory304pertaining to incident data120are ready for transmission and deleted once they are uploaded to the server123in order to preserve space on the memory104. The transmission state305typically defaults to the AI state301once it completes its functions, or in the case that no incident data120is stored in the memory104, or in the event that network connectivity is unavailable, the system defaults to the AI state301in which the system10,10′ can be configured to return to the transmission state305based on a schedule/timer in order to check network availability and whether incident data120is ready for upload.

It is recognized that multiple states/processes (some or all of the processes shown inFIG.15) can be accessed at the same time using separate processes/threads. It is recognized that storing and/or sending of the incident data120can occur in real time while the device101continues collection and processing of additional images117, sensor data118and metadata119. Preferably, the acquired incident data120can also be transmitted in a real time basis and/or scheduled basis, dependent upon appropriate network connectivity between the device101and the server123. Accordingly, sending incident data120over the network121can include queuing a set of incident data120in volatile memory103prior to subsequent transmission to the server123. Selected portions of incident data120are transmitted over the network121by the device101over to a server123for subsequent storage on a database202and file system204, for processing, storage access and reporting.

It is also recognized thatFIG.15helps to illustrate the different states in which the device101can function, but due to the multi-threaded/multi-process method in which this technology is implemented in practice, more back and forth communication can take place between different processes/states, and more processes/states may be present.

In a typical cloud based deployment, the server123acts as a gateway to users of the system10,10′, to make incident data120available in a meaningful and intuitive manner by way of accessing the data120transmitted to the server123. It is recognized that as one embodiment, the incident data120containing information of the incidents of interest12, can be portions of images117. Alternatively, or in addition to, the resultant processed data120can also include parts or all of the metadata119of the acquired incident images117.

Furthermore, the device101can upload incident data120intermittently when a connection is not accessible. For example, the device101can include non-volatile storage104that can store the incident data120temporarily and when a connection is available, send the data120through the network121to the server123. Some users may choose to only use wireless LAN connectivity in order to forgo cellular networks to save on cellular data costs. In this case, acquired incident data120and images117would be stored until the device101has access to a wireless access point connected to the internet. The device101can be configured to transmit data120at a scheduled rate, in order to benefit from improved performance, power consumption, and/or heat management.

Referring toFIG.16, shown is a flow diagram illustrating a workflow inclusive of the incident12acquisition process as well as clarifying the data19′ discarding process. The image acquisition logic402pertains to the method or processes, typically deployed as software108instructions, for collected image data117from the device's101camera107. Typically, images117are collected from the camera107on a periodic basis where the frequency of collection is usually limited by the device's camera107sensor,101CPU106and/or GPU110capabilities, as the vehicle102atravels along the road14. The aforementioned image acquisition logic402may be connected to two different workflow paths, related to manual incidents401(activating the incident acquisition process803a) or autonomous AI404analysis (using the process301,307). Typically, under normal operating conditions as per the autonomous process301, the collected images117continue through to the AI image processing404task, where the AI inferencing operations404(utilizing the AI modelling905) can be configured to generate identify incidents12in the images117, add labels automatically307, and optionally identify and discard406data (e.g. generate discard data19′ as discussed above) from the images117. Depending on the software's118programming, the discard data19′ may be redacted from the image117, or the whole image117may be redacted, or the entire image117may be regarded as discard data19′. For example, if there is no object of interest12in the image117, the image117can be discarded without being saved and transmitted.

In the event the entire image117is not discarded406, the image data117and resultant data118,119continues to the next packaging task408where it can be packaged with other relevant data. Other relevant data can include data related to the recently acquired image117such as relevant sensor data118that can include, for example GPS/GNSS coordinates, data pertaining to direction of travel and other sensor data118that may provide useful information of an incident12. Metadata119may be included in the packaging process408, and examples may include date and time information, incident label/tag and other relevant data that may provide useful information of an incident12. Once the relevant data pertaining to an incident12has been packaged408in a server123friendly format, the incident data120can be stored on the device101in a file system or database and/or sent to the server123for long term storage and access.

Other workflows may be utilized by the aforementioned system operation in order to provide further functionality. Such as, in the event that a trigger124or voice signal116event401is detected by the software108. Rather than to continue to process the (e.g. automatically) acquired image117using the AI processing404, the image117can be directly passed to the data packaging408portion of the workflow, where it is packaged, then stored/sent to the server123as incident data120.

There may be cases where images117acquired in this manner require image processing404in order to redact data19′ pertaining to personally identifiable information that may be present in a scene12, or provide further analysis using software108instructions. The system10′,10is designed to typically be operated in a vehicle102atravelling along public roads14. Therefore, captured images117can contain private information of citizens such as, faces and/or vehicle license plate numbers12. In some embodiments entire vehicles12and/or people12can be redacted406from an image117.

Therefore, in certain embodiments, a determining whether AI processing is required407may be included to route the acquired image117to the AI processing task404in order to redact personally identifiable image. In some embodiments, the AI image processing404task may include alternate processing instructions in order to accommodate this workflow.

Under normal operating conditions, the image processing404task may discard the image117entirely, however, due to the manual person trigger it may be processed in a different manner. Whereas an alternate set of image processing instructions contained in the AI processing task404may be included in order to process acquired images117obtained from the trigger/voice401functionality. Where this set of instructions can typically be configured to only redact personally identifiable information from an image117and discarding406it. In some embodiments, the aforementioned requirements may be achieved by configuring the image processing task to accept inputs from the trigger/voice401outputs. In the event that a trigger/voice signal401is detected and image processing404is required, the task404can be configured to not discard406entire images117and to only redact portions of the image data117. In some embodiments, the image processing operations for images117acquired by trigger/voice401functions may be deployed on the server123in the form of server side processing instructions203. Where the images117are processed subsequent to being uploaded409to the server123.

FIG.17illustrates an example data flow501scheme utilized by the system10,10′ to collect, assemble, store and send data120pertaining to incidents of interest12. The device101inputs to the system10,10′ come in the form of data such as images117and sensor data118, collected from the device's101camera107and sensor105modules. The image data117and sensor data118can be queued/stored into image data buffer, sensor data buffer osr simply fetched/polled upon demand.

The sensor data acquisition504and image acquisition503processes are typically low level and done by the device's101embedded hardware and software108. It is recognized that an embedded device101may have limited data acquisition capabilities—for example, limited frame rate or limited sensor refresh rate. It is also recognized that bottlenecks in the device's hardware101, including CPU106, GPU110, camera107, sensors105, Non-Volatile Memory (104) and volatile memory (103) can inhibit the device101from processing all the collected data117,118,119. There can also be processing delay in the underlying software108which can further limit the ability to process all data.

As such, in typical implementation, the software108would typically fetch the most recent image117and the most recent sensor data118based on the device's101processing ability. As such, some images117and associated sensor data118can be dropped.

The data flow501can include processing the image data117using AI image processing instructions905. The results from the AI processing905provide descriptive output data119in various forms for each image117, which can be used to determine whether the image data117contains incidents of interest12. In the case of an incident12, the AI905and related software108have the functionality to automatically to invoke the save, assemble, and send logic of the workflow408. The sensor data118associated with the image117is automatically fetched504, saved, assembled and sent with the image data117in the incident data102.

Alternatively, the incident packaging logic408(seeFIG.16) can be invoked with the use of an external trigger device124, or voice command116. The trigger device112transmits a trigger signal124to the device101, and includes data (e.g. label130) from indicating the type of incident being obtained. This data130is forwarded to the packaging logic408where it is assembled with the image117and sensor data118along with various other data pertaining to the incident119. The incident data120can be packaged for transmission to the server123.

Shown inFIG.16is further details of the incident acquisition process803aas initiated by the manual acquisition process signal(s)116,124, and the autonomous acquisition process307. It is recognized that any of the processes described in the figure can occur in parallel and/or in series.

For example, in the case where an image117(captured the camera107) is noted by the operator (e.g. seeing the captured image117on the screen111—seeFIG.13), the operator can send a command signal116,124in order to invoke the interactive process803afor the noted image117. For example, the command116,124can be used by the operator to add additional labels130(and any other descriptive metadata119) manually to the captured image117, such that these additional labels130(e.g. descriptive material) may or may not have been noted under the autonomous process307(e.g. as resulting from the processing404). One example trigger operation is where the operator notices a larger than normal pothole12in the image117and wants to make special mention of its location (and potentially other information such as size, etc). As discussed, the additional label(s)130can be selected via a list, as provided to the operator by the further processing function407of the system software108as invoked by the particular trigger signal/command116,124.

As such, it is recognised thatFIG.16shows multiple triggers116,124,307, for incident acquisition process803athat can be applied to captured images117, such that the results of the incident acquisition process803aare then sent to the packaging module408for assembly of the incident data120for sending to the server123. For example, one case is where for a particular captured image117, at task403the software108recognizes no/lack of presence of a command116,124(i.e. the particular image117is not flagged for interaction) and therefore the software108uses the AI process301(utilizing the AI methodology905) to identify and label307incidents12and incorporate any resultant metadata119and sensor data118as discussed above. Once processed, the task405can be used to send the data117,118,119(including identification of incidents12) to the packaging task408for assembly of the incident data120for communication over the network121to the server123(e.g. accounting for defined reporting intervals, address location of server123on the network121, server requests for data, etc.). As such, it is recognised that the packaging task408is used by the device101to assemble and send the incident data120in the predefined reporting frequency/content format as expected by the server123. It is recognized that in some embodiment, one process803awill handle all incident acquisitions, automated307and manual116,124, whereas in other embodiments, there may be different variations/multiple instances/modules of the incident acquisition process803afor different incident initiation options116,124,307which may be manual or automated.

It is recognised that the captured image117can be such an image117as automatically produced by autonomous operation of the camera107or otherwise requested by a command116,124. For example, the captured image117can be the result of the command116,124requesting that the camera107take an image of a particular section of the roadway14. In this case, the requested image117(as per the command116,124) can utilize the processing task407to process the image117using the AI process301before or after the process803ais performed (e.g. interactive adding of label(s)130by the operator via the command(s)116,124). For example, at task407, the device101can be instructed by the operator (via commands)116,124) to add the selected label(s)130interactively and them to simply pass the resultant image and label(s)130to the packaging task408. Alternatively, at task407, the device101can be instructed by the operator (via commands)116,124) to add the selected label(s)130interactively and them to pass the resultant image and label(s)130to the AI processing task404for AI processing (using the AI modeling905) and eventual receipt by the packaging task408. In this manner, the process803ais applied first and then the AI process301and labeling307is applied second.

In view of the above, it is recognized that the device101can implement the some or all of the processes116,124,301,307,803ain a number of different ways and in different order of operations. In terms of an image117autonomously captured by the camera107(e.g. not directed to specifically image capture via a command116,124received by the camera operational logic):1) for “no” presence403of a command116,124, the processes404,405, and408are utilized (and optionally406) to result in the incident data120sent to the server123;2) for “yes” presence403of a command116,124, the process803a(i.e. performing as an interactive image117) is utilized to add label(s)130before processes404,405and408can be performed to result in the incident data120sent to the server123(e.g. process404can be skipped on the device101for a particular interactive image117); and3) for “yes” presence403of a command116,124, the process803a(i.e. performing as an interactive image117) is utilized to add label(s)130after processes404,405(and optionally406) are performed, in order to affect the content of the incident data120before sending to the packaging task408(e.g. process404is used on the device101for a particular interactive image117).

Further to the above, it is recognised that for an interactive image117, the system10,10′ can continue to autonomously (e.g. using processes301,307) to acquire, process and send images117/incident data120reflective of the road14as the vehicle102ais operated/travelling.

Further to the above, it is recognised that for an interactive image117, the system10,10′ can be paused (i.e. autonomous process301,307is interrupted) until the interactive image117using the interactive process803ais acquired and processed to generate the interactive images117/incident data120reflective of the road14commanded116,124by the operator as the vehicle102ais operated/travelling.

The trigger/voice command activation process803acan be used as an alternative to the AI processing905workflows301. In this event, the latest image117and sensor data118are directly pulled from their respective buffers and assembled with the trigger data124and other relevant data commonly referred to as metadata119. Assembly typically includes packaging the image117, sensor118, and meta data119together in a file format suitable for server123communication. Suitable file formats can include XML, JSON or other file formats. As previously mentioned, files are temporarily saved to the device101in the case that network communication is unavailable, in which communication will be attempted at a later time.

Voice command116functionality can be included as an alternative to trigger device functions124in order to invoke the package-to-send logic408directly. Voice commands116are sent by the user and are typically in the form of an activation phrase116,124that the system10,10′ is configured to listen for. The speech-to-text data505(also referred to as label130) which includes utilizing the devices microphone125and audio interface102in order to execute the task of adding the data130to the incident data120via the AI package to send logic408step. The functionality is similar to the trigger device124functionality, in that it can invoke the packaging logic408as well as forwarding portions of metadata119(e.g. as obtained as labels130) to be assembled with the rest of the data120. This portion of metadata119typically includes the incident command116,124, which can be a portion of the phrase included after an activation phrase. The activation command116,124can therefore also invoke the packaging logic408, and then the incident command116,124with associated label data130is assembled with the latest image117and sensor data118, as selected initially via the command116,124under interaction by the operator of the device101.

The data packaging process408may include in the incident data120one image117and related data118,119per incident. In another embodiment, the data packaging process408may include in the incident data120a series of images117, or a video clip117related to the incident, together with the related data118,119. In another embodiment, the data packaging process408may include in the incident data120one or more incident image(s)117and related data118,119, as well as a video clip117of the incident data.

The number of additional images117may be programmed based on a number of frames117before and/or after the incident trigger116,124was activated. It is recognized that the software108may store images117in memory99,104for a certain period of time or up to a certain data volume and as such, it would be possible to retrieve images117that were acquired by the device's101camera(s)500before the incident capture116,124was initiated.

The duration of the video clip117may be programmed based on a number of units of time (for example, seconds) before and/or after the incident trigger116,124was activated. It is recognized that the software108may store images117or video chunks117in memory99,104for a certain period of time or up to a certain data volume and as such, it would be possible to retrieve images117that were acquired by the device's101camera(s)500before the incident capture116,124was initiated.

Referring toFIG.18, the trigger module's112inputs114may consist of a control interface114that may comprise of one or more switches114, such as push button switches, toggle switches, rocker switches, membrane switch, capacitive or resistive touch sensors, or any other type of mechanically or electrically actuated switching mechanism. The contact interface inputs114may consist of multiple inputs, where each specific input601can be made identifiable using instructions executed by the microcontroller113prior to communication with the AI device101. For example, specific input114can be assigned a unique identifier601such as a number, string, or value. This identifier601can then be transmitted124by the trigger module112to the AI device101. The AI device101can then parse the received signal116,124, and associate the identifier601, selected interactive image117and sensor data118to an incident data package120. The identifier601can be further associated, on the device101, or the server123, with a unique tag/label603(e.g. from the possible labels130—seeFIG.13) from a set of predefined tags/labels602(e.g. label list). As part of the acquisition process, the device101or server123can then tag130the acquired incident image117and data118with the tag/label603associated with the identifier601, based on the activated trigger input114.

The assignment process from trigger inputs114to tags/labels603can be configured602by the user. For example, it can be configured602to assign the first identifier601on the trigger module112to a tag/label603such as “road hazard”. Therefore, when the first identifier601is provided by the trigger module112, i.e. as part of the command124, the AI device101initiates the interactive acquisition process803a, and associates the acquired image117and sensor data118with specified “road hazard” tag/label603, prior to any further processing (optional) and transmission over the network121to the server123. In general, the associated tags/labels603can be manually inputted by the user and assigned to any of the identifier(s)601of the trigger inputs114(e.g. commands116,124).

The communication interface115of the trigger module112can comprise of a wireless network interface controller115, and/or a wired interface network interface controller115. The interface115can be used to establish network communication with the AI device101, as well as transmit commands116,124to the AI device101in order to initiate the incident acquisition process. In the case of a wireless transmitter, the networking technology used to establish communication between the devices can consist of technologies such as, Radio, Bluetooth, BLE, Wi-Fi (IEEE 802.11), ZigBee, or other wireless communication protocols. In the case of a wired interface, the technology used to establish communication may consist of a variety of serial or parallel communication protocols such as, CAN, I2c, SPI, Ethernet, RS232, USB, 1-Wire or other serial communication protocols used to establish communication between embedded devices.

The trigger module112can comprise a display, such as an LCD screen, LED screen, OLED screen, segment display, ELD screen, or other types of displays used to display images, numbers, strings etc. The display can be used to feedback to the user in the form of visual notification(s), when an input114has been activated. The display can be used to display the incident tag/label602that is assigned to an input114when that input114has been activated. The trigger module's112display can be combined with the trigger module's112inputs114, into one module112that can perform both of these functions. For example, with the use of a touch screen display, the trigger module112can implement capacitive or resistive touch inputs, while displaying feedback information on the screen.

Further to the above, it is recognised that the functionality of the device101and associated process803acan be configured to facilitate when the operator is following the images117being recorded, the operator would be able to select an image117(or video portion) and then ask the system process803ato manually add descriptors/labels130to the image117or to otherwise redact or unredact portions. For example, an operator action114(button push or voice command) can grab the latest picture117in the sequence buffer and add digital information130to it. The digital information130can be pre-programmed (i.e. button 1—pothole) or chosen out of a list/tagged with text (i.e. hey rover—tag pothole), seeFIG.18and related description.

Further, the operator can also ask the system via command116,124to take a specific picture117, i.e. point the camera500to a selected incident or region. In this case, the autonomous incident detection307can be temporarily halted (optional as one embodiment) while the camera500is being directed interactively by the operator. Otherwise, a separate picture117can betaken by the operator, using a different device, and thus this separate picture117can be sent to the smart camera101and then the operator can interactively adjust the picture117using the capabilities of the smart camera101(e.g. one or both of the processes307,803a).

It is noted that in some cases, the camera500field of view can already cover most the field of view in front of the windshield, meaning that whatever the driver sees, the device101is also likely to be see. The AI operations301grabs pictures117individually and analyzes them307. When the interactive action803ais taken, either the (e.g. last) picture117in memory is used (i.e. the last picture117that was analyzed by the AI) or a new picture117is taken, as per preferred embodiments. That picture117is then tagged with additional descriptive information130—either pre-programmed or chosen out of a list/tagged with text, seeFIG.18.

Further, from incident12tagging perspective, the AI tagging incident process307looks for certain objects/incidents12and it can be an independent process from the operator tagging incidents using process803a. There may be some overlap in the incident12acquisition116,124,307types for the different processes in interacting with the interactive acquisition process803a. Once an incident12has been tagged by an operator, the whole image117can be labeled with an incident tag130,603. The AI may then process307the image117to identify and redact personal information (people/cars), and look for other types of incidents12and tag them. If it is missed on the device101, the server123can also analyze the image117redact the information.