Patent ID: 12232232

DETAILED DESCRIPTION

As used herein, an element or step recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural elements or steps, unless such exclusion is explicitly recited. Furthermore, references to “one embodiment” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features.

Before addressing each figure individually, a brief overview of an Intermediate Device Structure (IDS) network is provided. The IDS network is an urban infrastructure device solution comprising a community of like members configured to make living in the city safer, healthier, and friendlier. The IDS members are communicatively coupled and distributed throughout the public domain of a city. The IDS community members can also be communicatively coupled to at least one remote client. The IDS members can communicate with at least one: an individual and/or his or her mobile device, a stationary or moving vehicle, a municipal department, a banking/credit institution, and cloud computing resources, as will be discussed. Examples of IDS members used in a network are found in the patent documents cited in the Cross-Reference Section, for example U.S. Pat. No. 11,071,204. However, the IDS member described herein has modified functionality to provide the functions and services as discussed herein.

The IDS member is coupled to an elevated structure that has or can have access to electricity. The elevated structures can include roadway lighting poles, traffic light poles, walls, power poles, and building exteriors. Coupling the IDS member to vertical structures well above grade enables greater area coverage to coupled sensing and communication devices, as well as providing a better vantage point to capture images with an image sensor which are used in black ice detection. Also, mounting IDS members well above the ground reduces risk of damage, tampering or theft.

Roadways constitute the majority of the city's public domain. Therefore, the IDS members can be primarily placed on roadway lighting and traffic lighting poles. These poles are electrified and the spacing between the poles is regulated, leaving no gaps that would otherwise require installing vertical structures to support the IDS members. The IDS member can incorporate the utility of the device/s originally coupled to the pole or can operate independently.

The IDS member's electronic devices can include at least one of: a processor/controller with resident memory and code, a sensing device, a communication device, a back-up power storage device, and an output device. Each of the IDS' networked members is tasked to gather in real time environmental inputs around the vicinity of the IDS member. Using sensing devices (e.g., image sensors, ground and air temperature sensors, humidity sensors, light sensors and the like) the information is received by the IDS' processor/controller. The sensing devices can be coupled to at least one of: the IDS' member housing, the pole/arm, and to a surface in the vicinity of the pole.

The sensing devices can include: a camera (which at least includes an image sensor and optics, and which can be a still camera or a video camera), a photocell, a temperature probe, a barometric pressure probe, a vibration sensor, a speaker/microphone, a light source, a wind velocity probe, an air quality probe, a radiation sensor, radar, a sound meter and any other sensing device that, operating alone and/or with other coupled sensing devices, enhances the utility derived from the IDS member.

The processor/controller of the IDS member is configured to receive and process a plurality of inputs in real time. The inputs are received from sensory devices associated with the IDS member, neighboring like IDS members and other remote clients. The IDS member can have a unique address and the associated sensing, communicating, output, and power back-up storage devices can be associated by a sub-address. Similarly, other remote IDS members, their devices, as well as remote clients have their own unique addresses and sub-addresses. Unique addresses of IDS member devices enable geographic mapping of the IDS' member community.

The code operating the processor/controller of the IDS member is configured to operate alone and/or in unison with another networked member. The code can employ at least one artificial intelligence (AI) algorithm embodied as a trained AI engine, as will be discussed in more detail with reference toFIGS.11,12and13. AI algorithm modules that can provide I/O and/or control an IDS member include at least one of: a learning and a predictive algorithm module. The inputs received by the AI code can include sensory input that can perceived by humans, and other inputs beyond human perception.

The inputs received by the devices associated with the IDS member coupled with inputs received from like IDS neighboring members and other remote clients is compiled by the code's pre-programmed parameters to generate accurate and consistent outputs based on the same sets of inputs. The outputs can be preemptive or reactive. The IDS community of networked members' prime responsibility is to protect life. Applicant's prior U.S. Pat. Nos. 9,829,185; 9,885,451; 10,215,351; 10,653,014; and 11,071,204, each of which is incorporated herein by reference in its entirety, articulate several utility use cases.

The present embodiments describe, among other things, use cases where an IDS member can protect life under intermittent environmentally hazardous conditions. In this case, a phenomenon known as black ice is occasionally unpredictable. Knowing when such events materialize is critical to preserving life.

With reference toFIG.1, at least one black ice sensing device18is/are mounted to vertical structure/s well above a vehicles' height215. Along roadways3, the sensing device/s18is/are coupled to the IDS member10. The sensing device/s18can also couple to roadway lighting poles5and traffic light poles4that support the IDS member10. The electrified roadway lighting poles5and traffic light poles4enable placing the IDS member10sufficiently high for the sensing device/s18(e.g., imaging devices such as cameras, and air temperature sensors) to have full coverage of roadways3and/or pedestrian100pathways. It should be noted that other sensors such as temperature and/or humidity sensors may be placed near, or at, the bottom of the poles5/4, or adjacent to the road surface, so as to obtain sensor data in the micro region around a the surface of the road.

In Applicant's prior patents (cited above), the disclosures describe a camera60(the camera60may capture video images as well as, or in addition to, still images) communicatively coupled to an IDS member's10processor/controller. The camera60has an assigned a field of coverage referred herein as a zone section11. In some cases, the field of coverage can overlap other neighboring IDS members'10fields of coverage. Together, the cameras communicatively coupled to the community of IDS members10have full coverage of roadways3and/or pedestrian100pathways. The coverage also includes stationary and mobile objects such as vehicles215and pedestrians100.

These earlier patents further taught that the processor/controller can be taught to pixelize its field of vision to discern anomalies from the expected field of coverage. The camera60communicatively coupled to the IDS member10can have at least one filter that filters refracted light in the visual and non-visual spectrum to humans. The input (e.g., image data, as well as optional position, and time stamp data) captured by the camera60is processed through at least one of: the camera's60image processor and/or the processor/controller of the IDS member10, so as to detect when black ice1has formed or will soon form. The camera60of the IDS member10can operate 24 hours a day, 7 days per week, recording and processing the same field of coverage day and night in all seasons. The data gathered is then stored by each IDS member10corresponding to the IDS member's10zone section11location. The IDS member10employs code to determine a change in refracted or reflected light properties at the surface of the roadway to detect or anticipate black ice that forms under a myriad of climatic conditions associated with seasonality and time of day. The refracted and reflected light properties are collected and stored in a relational database that includes associated values of temperature, humidity, ambient light level, refracted or reflected light level, change in refracted or reflected light level over a predetermined period of time (e.g., 10 seconds, 1 minute, 10 minutes), and optionally whether black ice under similar past data recordings. As will be discussed in reference toFIGS.11-13, the IDS members processor may also employ a trained AI engine to assist in detecting presence of black ice1. The amount of reflectivity change is in a range of 5% to 10%, although both the lower end and upper end are adjustable within a range of 1% to 20%.

The gathered data coupled with communicated input from neighboring IDS members10and remote client/s are compiled with local resident code program parameters to generate at least one output. The code may employ at least one AI algorithm, enhancing the IDS member's10ability to identify and alert detection of black ice1. The AI code can include input from at least one additional sensing device18other than the camera60, discerning the presence of, or the imminent possibility for black ice1to form. For example, the IDS member10processor/controller receiving input from the camera60may have observed water ponding along the path of vehicular215travel. A temperature probe communicatively coupled to the IDS member10processor/controller AI code tasked with monitoring fluctuations in the ambient air temperature provides data in real time of descending ambient temperature. With these two input parameters, the AI code can predictably decide that the presence of black ice1is imminent. In such cases, the AI code sends an alert to “need-to-know” clients (other electronic entities that are part of the IDS member network, including devices serviced by the IDS member network) in advance of the icing event. In another example, an audio device8and/or a light source9communicatively coupled to a traffic light pole4and/or to a roadway lighting pole5with coupled IDS member10in the vicinity of a pedestrian100crosswalk12can alert pedestrians of anticipated or present black ice1in the crosswalk12, and take appropriate measures to issue warning signals that trigger an audio, visual, and/or tactile alert for the pedestrian or a mobile device carried by the pedestrian.

The AI code may employ a learning module. Every time it records an event, the AI code extracts pertinent data points to help the code to at least become more efficient in predicting and/or identifying the presence of black ice1.

The networked community of the IDS members10communicates with one another on a programmed “need-to-know” basis. Since each IDS member's10geographic location is known, through cross-communication between the IDS members10, humans and machines can become aware of black icing1events long before arriving at the specific location of the black ice1. For example, at least one IDS member10identifies an imminent black icing1event about to occur in a specific location along its field of coverage (Zone section11). The IDS member10immediately communicates the imminent event to neighboring IDS members10. The neighboring IDS members10identify vehicles215and/or pedestrians100traveling in the direction of the black ice1location, and by means of wireless and/or audio, alerts vehicles215and/or pedestrians100respectively.

Since the icing event location is known, and the vehicle/s215alerted is/are known, the vehicles'215distance to the black ice1location can be configured in real time, letting the vehicle/driver know the distance to the location and signaling “all clear” once the vehicle215passes the black ice1location. As will be discussed, once alerted, the vehicle may also serve as a detecting resource by controlling an illumination source and image capture device on the vehicle that is oriented under the vehicle to detect the presence/absence of black ice from a very close distance (e.g., 1 to 3 feet depending on the mounting height of the illumination source and image capture device on the vehicle). The vehicle215can thus confirm the presence (or absence) of black ice at very precise positions along the roadway, and share that information with the closest IDS member, or other device in the IDS member network such as another vehicle or a roadside device (RSD) via V2V or more generally V2X communication. Moreover, in this context the vehicle215does not perform the detection for its benefit because it detects the black ice1while the vehicle215is over top of the black ice1. However, the close-up detection of black ice1by the vehicle215provides highly accurate data that is shared with the IDS member network for the benefit of other vehicles or pedestrians who pass over the same section of pathway, which has black ice1that is present on it. Similarly, the vehicle215is equally instrumental in confirming that the black ice1is absent when conditions have changed and the black ice1returns to liquid or vapor form.

Once an alert condition is triggered, at least one municipal/county department can be alerted. The benefits of such an alert can include:Assessment where roadway/crosswalk3,12repair is neededPlacing emergency staff and equipment on alertAdding to code data such as traffic volume along any zone section11of a roadway3governed by an IDS member10and the corresponding frequency of accidents within the zoned section11can help decision makers determine the priorities of maintenance and repair work to avoid accidents caused by at least black ice1.

The means of communicating with vehicles215can be different from the means of communicating with municipal/county and/or pedestrian100clients. The communications industry has developed a wireless standard V2V and more generally V2X (V2V is a subset of V2X) communication standard used for wireless communications with moving vehicles, and V2X can be adopted by the network of the IDS members'100community. A different protocol can be used as well for communicating with municipal/county departments. Since proximity and/or speed do not factor when it comes to pedestrian100travel, public audio and/or visual devices8,9can be sufficient to alert pedestrians100in the vicinity of a black ice1presence.

FIG.1is a perspective view of a four-way intersection with a portion of its surface covered by black ice1. Inside the intersection four traffic light poles4shown control vehicular and pedestrian100traffic. Along the roadways3leading to the intersection7, a plurality of roadway lighting poles5are configured to illuminate the roadways3, intersection7, and sidewalks6.

FIG.1shows a surface area, within the intersection7identified by at least one IDS member10, to be covered by black ice1. The area is shown by an enclosed irregular oblong shape with diagonal lines across. Also shown is a vehicle215making a left turn through the intersection7traveling over the patch of the black ice1. The IDS members10are coupled to the roadway lighting poles5, usually at a height of more than 20 feet, which gives the IDS members10an excellent vantage point for observing the pathways. At least one sensing device18is communicatively coupled to the IDS member10. The sensing device18such as a camera60can couple to the IDS member housing15and/or a section of the pole2section. In addition, a device coupled to a nearby pole2, like a traffic light pole4, can be remotely controlled by an IDS member10.

Each of the coupled IDS members10is associated with a specific geographic zone section11of the roadway5and/or the intersection7. The sensing device/s18coupled are configured to know each pixelated area of the zone section11assigned to their IDS member11to oversee and manage. Each of the IDS members10has a unique address and the IDS member10associated coupled devices are assigned a sub-address of the IDS member10.

FIG.1shows the entire area of the four-way intersection7and the roadways3leading to the intersection7zoned by sections11. Each of the zone sections11is also associated with a specific IDS member10device tasked with at least one functionality of: monitoring, recording, managing, and alerting. The zone section11shown can include the IDS member10number and the associated zone section11, i.e., IDS65Z81. The patchwork of all zones provides 24/7 coverage over the entire city roadway system.

FIG.1shows lines of vision (lines of sight) extending from the pole2coupled IDS member10to a vehicle215passing through the intersection7. These lines represent the sensing device18of the IDS member10tracking a vehicle215driving over a patch of black ice1. At this point of travel both the driver and the vehicle215are most vulnerable to an accident. In some jurisdictions, at least one operation by the vehicle215may be controlled by an IDS member10when a vehicle215advances toward a road impediment such as a patch of black ice1.

The IDS members10shown separately and/or jointly can continuously monitor the size and shape of the black ice1patch and can relay the information automatically in real time and/or when requested to mobile and stationary clients. The sensing devices18identifying a black ice1patch location can have multiple utility. These devices can sense in real time man-made and natural environmental phenomena. At least one of the sensed inputs can be beyond human perception.

The key utility derived from the IDS member10community as shown inFIG.1is the networked system's ability to give humans and machines real time advance warning of intermittent events that are about to happen and enough time to prepare for, in the present use case an adverse road condition ahead.

FIG.2shows an elevation of a roadway3with IDS members10coupled to roadway lighting poles5communicating with a vehicle215and a pedestrian100in the vicinity of an area with black ice1. Two poles2, each with an IDS member10coupled, are shown ahead of the direction the vehicle215travels toward. At the poles'2middle section, a coupled signaling device9such as a light source13is configured to visually alert drivers when approaching a location identified to be covered by black ice1. The signaling devices9can be controlled by the IDS member'10processor/controller of the roadway pole5they are coupled to.

At the opposite end of the elevation, a traffic light pole4is shown with a pedestrian100crossing the roadway3at a crosswalk12. An audiovisual device8,9coupled to the traffic light pole4can be configured to alert pedestrians100when black ice1cover has been detected along a crosswalk12. The audio device8and/or a light source9can be controlled by an IDS member10coupled to the traffic light pole4, or a nearby IDS member10coupled to a roadway lighting pole5. The audio alert can be one of a plurality of audio messages communicated by an IDS member10processor/controller managing pedestrian100traffic in a crosswalk12.

FIG.2shows two lines extended between the two roadway lighting pole mounted IDS members10and the approaching vehicle215. These lines represent communication connectivity between the IDS members10and at least one vehicle1. The communication can be single or bi-directional. It is noted that as an example, the present figure V2V (or V2X) communication protocol can be adapted to communicate between pole mounted IDS members10and a vehicle215.

Of the two roadway lighting poles5, the IDS member10pole2shown closer to the black ice1patch displays a direct line to the center of the black ice1patch and another direct line to the pole2positioned next to the approaching vehicle215. The IDS member10coupled to the pole2closer to the black ice1patch maintains constant surveillance of the black ice1patch and on an “as needed” basis communicates the condition to the neighboring IDS member10coupled to the pole2in the vicinity of the vehicle215.

As a vehicle215nears the black ice1patch, it can be configured to communicatively interact with the IDS member10closest to the black ice patch1. Addressing roadway mitigation of roadway hazards, an onboard dashboard display inside the vehicle can inform the driver of the distance to the black ice patch, the patch configuration, and when the vehicle has cleared the black ice1patch.

FIG.3shows a functional block diagram that depicts one example of the IDS member process logic when detecting or predicting a black ice event. The processes are shown in a stepwise format.

Step 1—The IDS member sensing device/s sends a data set to the processor/controller.

Step 2—The processor/controller processing the data set identifies the presence of black ice at a specific zone section, or that it is imminent that black ice will be formed at a specific zone section within a time window.

Step 3—The IDS member sends an audio, a visual, or an audio/visual alert to pedestrian/s if they are in the vicinity of the black ice event.

Step 4—The IDS member sends electronic alert/s to oncoming vehicles.

Step 5—The IDS member sends electronic alert/s to neighboring IDS network members. The alert can identify the exact location of the zone section and the icing patch size.

Step 6—The IDS member sends electronic alert/s to municipal/county department/s identifying the section zone and the icing patch size.

Step 7—Neighboring IDS members turn on their signaling device such as a light source coupled to a pole, configured to alert oncoming drivers to the presence of black ice patch/es ahead. Vehicles communicatively coupled to the pole mounted IDS member receive in real time at least one data point on the specific zone section location of the black ice patch. The data points can include the black ice patch size and distance to the ice patch.

Step 8—The alerting IDS member and/or neighboring member/s (when the black ice patch extends over a plurality of zone sections) monitor vehicle/s until the vehicle/s clear the location of the black ice patch. Vehicle/s communicatively coupled to the IDS member can be notified once the vehicle/s have cleared the black ice patch.

Step 9—After the black ice has been removed/melted, the IDS member's processor code extracts data to add to the historical records and, when learning algorithms used, incorporates the data points into the learning algorithm for improving future performance.

Step 10—After the black ice is removed/melted, the IDS member's processor code sends a report to the municipality/county with historical data that can prioritize addressing issues occurring at the black ice zone section. The report can be isolated to only black ice events, or can include other events occurring at the specific zone section.

The steps shown above represent several essential steps; however, some of the steps can be deleted while other steps can be added. Further, the steps do not have to follow the same order and can occur concurrently.

FIG.4is a signaling diagram between (1) a detecting pole (e.g., a pole that hosts at an IDS member that has detected black ice, or anticipates that black ice will be formed at any moment), (2) a vehicle, and (3) a relay node, which may be another IDS member on a pole, or another vehicle on the road, or a V2X road-side unit (RSU). The RSU may be deployed near the roadway and connected to power from the electrical grid, or it may be more portable and autonomous by receiving its power locally from a battery. In response to the IDS member (hosted on the detecting pole) detecting black ice, the IDS member on the detecting pole dispatches a report301to a relay node either through an addressed message or a beacon signal so as to inform the relay node. In this embodiment, the vehicle registers its presence with a signal303with an IDS member, which in this embodiment is done directly with the relay node or another vehicle that is ahead of the vehicle. An authentication, or even an ACK,305is returned to the vehicle so the vehicle's processor knows it is within the IDS member network. The vehicle then shares its present route, and its planned route in message307with the relay node. The relay node then checks for a correspondence with vehicle's travel route and the black ice location, and in response reports any coincidence to the vehicle in a message309. In reply, the vehicle's processor may take corrective action, such as adjust its sensitivity to self-detection of black ice (previously discussed), adjust a speed, generate a warning, or automatically engage all-wheel drive mode, or anti-lock wheel mode once the vehicle approaches the zone with the black ice. The vehicle may also enter a relay mode itself, and dispatch a warning signal311to other vehicles, using V2X signaling, following in its path so the other vehicles are warned of the black ice ahead.

Subsequently, the vehicle may enter a black ice self-detect mode, where itself scans for the presence of black ice in the danger zone and provides update reports in step313. To enhance detectability, the vehicle may illuminate an undercarriage light to enhance the vehicle's onboard image sensor to detect a change in reflectivity of the road surface, as an indication/confirmation that black ice is present. The information gleaned by the vehicle as is crosses over the black ice, is directly communicated in another signal(s)315with one, or both, of the IDS member on the detecting pole and the relay node. The signaling also has an alert cancel signal317that informs the vehicle and the relay node if the black ice is determined to have changed back to a liquid or vapor state.

FIG.5is a user interface inside the vehicle. A display500includes a first display panel507, and a second display panel509. The first display panel includes user-selectable settings that allow for adjustments up or down regarding the sensitivity level of the vehicle's black ice self-detection mode. Each increment of adjustment adjusts the probability of detection by a predetermined amount such as 0.25% (or some other level) so the user has some influence on the false alarm rate for the vehicle's black ice self-detection mode. The right panel509provides a visual display of a roadway, and a location on the roadway (icon513) where the black ice has been detected by other IDS members.

FIG.6is a flowchart of process performed by a processor of an IDS member that detects a presence of black ice. The process begins in step S200where the IDS member engages temperature and humidity sensors to determine if black ice is plausible under detected conditions. The sensors also include image sensors that observe changes in reflectivity of the road surface with black ice being detected with a rapid and noticeable increase in reflectivity. In the case of the IDS member being mounted to a streetlight, the streetlight illuminates the roadside surface so the presence of black ice is more noticeable when the reflection angle from the streetlight is closer to orthogonal than when illuminated by headlights of a vehicle, which are oriented to have a more of a glancing angle (e.g., closer to 180 degrees than 90 degrees) off the roadside surface than from an overhead light. The processor may also employ a locally hosted AI engine that has been trained on images from overhead perspectives, or a remotely located AI engine (e.g., hosted on a cloud network, and in communication with the IDS member).

If black ice is not detected in S202the process returns in S203to step S200. However, when black ice is detected in S202, the process proceeds to S204where the IDS member that detects the black ice transmits a beacon signal to other IDS members as well as vehicles and relay nodes. Furthermore, the IDS member reports the existence of the black ice to authorities in S205so authorities can take municipal or state action by triggering traffic control signals and other visual or auditory warning signals that are directly apparent to an individual (e.g., visual or auditory signals), or via a mobile device in the person's possession, or via the user interface equipment in vehicles. If the IDS member does not detect, in S206, a presence of any local vehicles, the IDS member continues to broadcast a warning beacon. On the other hand, if a vehicle is present, the IDS member establishes direct communications, or communications with a relay node, in S207until the vehicle is clear of the black ice. Subsequently, after the black ice is no longer detected, the IDS member in S208provides a comprehensive report to local and/or state authorities so the authorities can accumulate data and detect patterns of where black ice usually form, and thus have a motivation to take corrective action at those locations.

FIG.7is a flowchart of a process followed by a vehicle's processor when driving in the IDS member network. The process begins in S300where the vehicle monitors IDS message traffic. The in S301, the vehicle receives a message, and checks in S302whether the IDS message indicates black ice is nearby. If not, the process returns in S303to continue monitoring in S300. However, if the received message indicates black ice is detected the vehicle's processor checks in S304for the location of the black ice, and in S305engages the vehicle's safety measures (e.g., throttle control, AWD mode, etc.) as well as optional engage a self-detect black ice mode. Then in S306the processor inquires whether the black ice exists where it was reported to exist, and if so continues the safety measures of S305. However, the if the self-detect mode does not reveal black ice, or the vehicle has passed the black ice, the vehicle reports to the IDS member network its interaction/detection in S307before resuming driving in S308.

FIG.8is a block diagram of circuitry used for wireless communications in the vehicles, IDS members, relay nodes, as well as other devices employing RF communications described herein, such as the RSU used in V2X communication with the IDS member and the vehicles. In the example ofFIG.8, the communications in a RSU will be used as a non-limiting example. The RSU includes processor circuitry (e.g., programmed computer)805, a radio front end263, an antenna, a tactile and visual interface267, permanent memory, memory card678, an optional lithium ion battery264, timer279, and location circuitry266. The RSU may include any sub-combination of the components described above.

The processor805may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor805may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the RSU to operate in a wireless environment, with or without grid power. The processor805may be coupled to the radio front end263, which may be coupled to the antenna262. WhileFIG.10shows the processor805as a separate component, the radio front end may be integrated with the processor805they may be integrated into an electronic package or chip.

The antenna262may be a whip or a patch antenna, or may be an array of whip and patch antennas along with phase shifting circuitry so as provide gain and perform beam steering. In turn, the beam steering may be advantageous in directing the transmit energy to a nearest neighboring relay node, thereby allowing for the communication link to be closed at great distances.

The radio front end263may be configured to modulate the signals that are to be transmitted by the antenna262and to demodulate the signals that are received by the antenna262. The processor805of the transceiver212may be coupled to, and may receive user input data from, the sensors of the IDS member. The processor805may access information from the permanent memory130and/or the memory card678. The permanent memory130may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The memory card678may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. The processor805may access information from, and store data in, memory that is not physically located on the RSU.

The processor805may receive power from the lithium ion battery264, or other power source, such as a wired power supply, etc., and may be configured to distribute and/or control the power to the other components in the RSU.

The processor805may also be coupled to a GPS location circuitry136, which provided location information (e.g., longitude and latitude) regarding the current location of the RSU. In addition to the location circuitry136, the RSU may receive location information via wireless signal232from the IDS member.

The timer279is a programmable timer that includes a clock. It operates under direction of the processor805, and serves as a wake-up timer so the RSU (which may be a mobile device and deployed when or where needed) can enter sleep mode, and then be woken up the by the timer at determined times to check temperature. If the temperature is well above freezing, the RSU can enter a sleep cycle until woken again by the timer. The main purpose of the timer is to allow the RSU to remain in a sleep state so as to conserve battery power, and then only wake up occasionally to take temperature measurements, and then operate at freezing or below freezing temperatures.

FIG.9is a block diagram of an exemplary IDS system in which one or more apparatuses of the IDS network may be used to communicate via V2V or V2X communications. 5G or other wireless communications may be used that support the V2V and V2X communication protocols. For example, as shown inFIG.9, a gNode B161may serve as a base station to service all communication devices within its range. However, some vehicles that are part of a V2X group may not be covered within the cell's territory, but could relay black ice messages via themselves using V2V communications, or via the RSU163, which is deployable in the field where needed. Furthermore, the vehicles may also communicate directly or via a V2X server160that serves as a hub to assist in V2V/V2X communications.

FIG.10illustrates a block diagram of a computer that may implement the various embodiments described herein. The present disclosure may be embodied as a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium on which computer readable program instructions are recorded that may cause one or more processors to carry out aspects of the embodiment.

The computer readable storage medium may be a tangible device that can store instructions for use by an instruction execution device (processor). The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any appropriate combination of these devices. A non-exhaustive list of more specific examples of the computer readable storage medium includes each of the following (and appropriate combinations): flexible disk, hard disk, solid-state drive (SSD), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash), static random access memory (SRAM), compact disc (CD or CD-ROM), digital versatile disk (DVD) and memory card or stick. A computer readable storage medium, as used in this disclosure, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Computer readable program instructions described in this disclosure can be downloaded to an appropriate computing or processing device from a computer readable storage medium or to an external computer or external storage device via a global network (i.e., the Internet), a local area network, a wide area network and/or a wireless network. The network may include copper transmission wires, optical communication fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing or processing device may receive computer readable program instructions from the network and forward the computer readable program instructions for storage in a computer readable storage medium within the computing or processing device.

Computer readable program instructions for carrying out operations of the present disclosure may include machine language instructions and/or microcode, which may be compiled or interpreted from source code written in any combination of one or more programming languages, including assembly language, Basic, Fortran, Java, Python, R, C, C++, C# or similar programming languages. The computer readable program instructions may execute entirely on a user's personal computer, notebook computer, tablet, or smartphone, entirely on a remote computer or computer server, or any combination of these computing devices. The remote computer or computer server may be connected to the user's device or devices through a computer network, including a local area network or a wide area network, or a global network (i.e., the Internet). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by using information from the computer readable program instructions to configure or customize the electronic circuitry, in order to perform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flow diagrams and block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood by those skilled in the art that each block of the flow diagrams and block diagrams, and combinations of blocks in the flow diagrams and block diagrams, can be implemented by computer readable program instructions.

The computer readable program instructions that may implement the systems and methods described in this disclosure may be provided to one or more processors (and/or one or more cores within a processor) of a general purpose computer, special purpose computer, or other programmable apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable apparatus, create a system for implementing the functions specified in the flow diagrams and block diagrams in the present disclosure. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having stored instructions is an article of manufacture including instructions which implement aspects of the functions specified in the flow diagrams and block diagrams in the present disclosure.

The computer readable program instructions may also be loaded onto a computer, other programmable apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions specified in the flow diagrams and block diagrams in the present disclosure.

FIG.10is a functional block diagram illustrating a networked system800of one or more networked computers and servers. In an embodiment, the hardware and software environment illustrated inFIG.10may provide an exemplary platform for implementation of the software and/or methods according to the present disclosure.

Referring toFIG.10, a networked system800may include, but is not limited to, computer805, network810, remote computer815, web server820, cloud storage server825and computer server830. In some embodiments, multiple instances of one or more of the functional blocks illustrated inFIG.10may be employed.

Additional detail of computer805is shown inFIG.10. The functional blocks illustrated within computer805are provided only to establish exemplary functionality and are not intended to be exhaustive. And while details are not provided for remote computer815, web server820, cloud storage server825and computer server830, these other computers and devices may include similar functionality to that shown for computer805.

Computer805may be a personal computer (PC), a desktop computer, laptop computer, tablet computer, netbook computer, a personal digital assistant (PDA), a smart phone, or any other programmable electronic device capable of communicating with other devices on network810.

Computer805may include processor835, bus837, memory840, non-volatile storage845, network interface850, peripheral interface855and display interface865. Each of these functions may be implemented, in some embodiments, as individual electronic subsystems (integrated circuit chip or combination of chips and associated devices), or, in other embodiments, some combination of functions may be implemented on a single chip (sometimes called a system on chip or SoC).

Processor835may be one or more single or multi-chip microprocessors, such as those designed and/or manufactured by Intel Corporation, Advanced Micro Devices, Inc. (AMD), Arm Holdings (Arm), Apple Computer, etc. Examples of microprocessors include Celeron, Pentium, Core i3, Core i5 and Core i7 from Intel Corporation; Opteron, Phenom, Athlon, Turion and Ryzen from AMD; and Cortex-A, Cortex-R and Cortex-M from Arm.

Bus837may be a proprietary or industry standard high-speed parallel or serial peripheral interconnect bus, such as ISA, PCI, PCI Express (PCI-e), AGP, and the like.

Memory840and non-volatile storage845may be computer-readable storage media. Memory840may include any suitable volatile storage devices such as Dynamic Random Access Memory (DRAM) and Static Random Access Memory (SRAM). Non-volatile storage845may include one or more of the following: flexible disk, hard disk, solid-state drive (SSD), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash), compact disc (CD or CD-ROM), digital versatile disk (DVD) and memory card or stick.

Program848may be a collection of machine readable instructions and/or data that is stored in non-volatile storage845and is used to create, manage and control certain software functions that are discussed in detail elsewhere in the present disclosure and illustrated in the drawings. In some embodiments, memory840may be considerably faster than non-volatile storage845. In such embodiments, program848may be transferred from non-volatile storage845to memory840prior to execution by processor835.

Computer805may be capable of communicating and interacting with other computers via network810through network interface850. Network810may be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and may include wired, wireless, or fiber optic connections. In general, network810can be any combination of connections and protocols that support communications between two or more computers and related devices.

Peripheral interface855may allow for input and output of data with other devices that may be connected locally with computer805. For example, peripheral interface855may provide a connection to external devices860. External devices860may include devices such as a keyboard, a mouse, a keypad, a touch screen, and/or other suitable input devices. External devices860may also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present disclosure, for example, program848, may be stored on such portable computer-readable storage media. In such embodiments, software may be loaded onto non-volatile storage845or, alternatively, directly into memory840via peripheral interface855. Peripheral interface855may use an industry standard connection, such as RS-232 or Universal Serial Bus (USB), to connect with external devices860.

Display interface865may connect computer805to display870. Display870may be used, in some embodiments, to present a command line or graphical user interface to a user of computer805. Display interface865may connect to display870using one or more proprietary or industry standard connections, such as VGA, DVI, DisplayPort and HDMI.

As described above, network interface850, provides for communications with other computing and storage systems or devices external to computer805. Software programs and data discussed herein may be downloaded from, for example, remote computer815, web server820, cloud storage server825and computer server830to non-volatile storage845through network interface850and network810. Furthermore, the systems and methods described in this disclosure may be executed by one or more computers connected to computer805through network interface850and network810. For example, in some embodiments the systems and methods described in this disclosure may be executed by remote computer815, computer server830, or a combination of the interconnected computers on network810.

Data, datasets and/or databases employed in embodiments of the systems and methods described in this disclosure may be stored and or downloaded from remote computer815, web server820, cloud storage server825and computer server830.

Circuitry as used in the present application can be defined as one or more of the following: an electronic component (such as a semiconductor device), multiple electronic components that are directly connected to one another or interconnected via electronic communications, a computer, a network of computer devices, a remote computer, a web server, a cloud storage server, a computer server. For example, each of the one or more of the computer, the remote computer, the web server, the cloud storage server, and the computer server can be encompassed by or may include the circuitry as a component(s) thereof. In some embodiments, multiple instances of one or more of these components may be employed, wherein each of the multiple instances of the one or more of these components are also encompassed by or include circuitry. In some embodiments, the circuitry represented by the networked system may include a serverless computing system corresponding to a virtualized set of hardware resources. The circuitry represented by the computer may be a personal computer (PC), a desktop computer, a laptop computer, a tablet computer, a netbook computer, a personal digital assistant (PDA), a smart phone, or any other programmable electronic device capable of communicating with other devices on the network. The circuitry may be a general purpose computer, special purpose computer, or other programmable apparatus as described herein that includes one or more processors. Each processor may be one or more single or multi-chip microprocessors. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. The circuitry may implement the systems and methods described in this disclosure based on computer-readable program instructions provided to the one or more processors (and/or one or more cores within a processor) of one or more of the general purpose computer, special purpose computer, or other programmable apparatus described herein to produce a machine, such that the instructions, which execute via the one or more processors of the programmable apparatus that is encompassed by or includes the circuitry, create a system for implementing the functions specified in the flow diagrams and block diagrams in the present disclosure. Alternatively, the circuitry may be a preprogrammed structure, such as a programmable logic device, application specific integrated circuit, or the like, and is/are considered circuitry regardless if used in isolation or in combination with other circuitry that is programmable, or pre-programmed.

The IDS member may employ a trained AI engine to assist in detecting black ice formation, and assist in predicting when ice will become present based on emerging environmental factors. As shown inFIG.11, a computing device1000may include a data extraction network2000and a data analysis network3000. Further, as illustrated inFIG.12, the data extraction network may include at least one first feature extracting layer210, at least one Region-Of-Interest (ROI) pooling layer220, at least one first outputting layer230and at least one data vectorizing layer240. And, also to be illustrated inFIG.13, the data analysis network3000may include at least one second feature extracting layer310and at least one second outputting layer320.

First, the computing device1000is trained on images and sensor information provided to it by the sensors and camera(s) of the IDS member. After a subject image is acquired, in order to generate a source vector to be inputted to the data analysis network3000, the computing device1000may instruct the data extraction network2000to generate the source vector including (i) a reflectivity of the roadway's surface, and (ii) an estimated presence of black ice on the road surface.

In order to generate the source vector, the computing device1000may instruct at least part of the data extraction network2000to detect reflectivity and black ice presence from the image data from the IDS member.

Specifically, the computing device1000may instruct the first feature extracting layer210to apply at least one first convolutional operation to the subject image and sensor data, to thereby generate at least one subject feature map. Thereafter, the computing device1000may instruct the ROI pooling layer220to generate one or more ROI-Pooled feature maps by pooling regions on the subject feature map and/or sensor data, corresponding to ROIs on the subject image, and/or senor data file which have been acquired from a Region Proposal Network (RPN) interworking with the data extraction network2000. And, the computing device1000may instruct the first outputting layer230to generate at least one estimated reflectivity. That is, the first outputting layer230may perform a classification and a regression on the subject image and sensor file, by applying at least one first Fully-Connected (FC) operation to the ROI-Pooled feature maps, to generate each of reflectivity and black ice formation detection, including information on coordinates of each of bounding boxes on a specific area around particular roadway (or pathway) regions that are traversed by vehicles or pedestrians.

After such detecting processes are completed, by using the estimated reflectivity and black ice formation detection, the computing device1000may instruct the data vectorizing layer240to subtract a y-axis coordinate of an upper bound of the ground from a y-axis coordinate of the lower boundary of the region surrounding probe to generate the apparent reflectivity and ice detection associated with the content sensor file from region of the roadway, and multiply the detected value with an estimated area to generate the apparent reflectivity and black ice presence for that area.

After the apparent reflectivity and black ice formation for the area is acquired, the computing device1000may instruct the data vectorizing layer240to generate at least one source vector including the reflectivity and estimated ice presence as its at least part of components.

Then, the computing device1000may instruct the data analysis network3000to calculate an estimated ice presence by using the source vector. Herein, the second feature extracting layer310of the data analysis network3000may apply second convolutional operation to the source vector to generate at least one source feature map, and the second outputting layer320of the data analysis network3000may perform a regression, by applying at least one FC operation to the source feature map, to thereby calculate the estimated ice presence.

As shown above, the computing device1000may include two neural networks, i.e., the data extraction network2000and the data analysis network3000. The two neural networks should be trained to perform said processes properly. Below, how to train the two neural networks will be explained by referring toFIG.12andFIG.13.

First, by referring toFIG.12, the data extraction network2000may have been trained by using (i) a plurality of training images and sensor measurements corresponding to road zones for training, photographed from above the road zones by the IDS members, and (ii) a plurality of their corresponding GT Ice reflectivity and ice detections. More specifically, the data extraction network2000may have applied aforementioned operations to the training images and sensor files, and have generated their corresponding estimated regions and estimated reflectivity and ice presence amounts. Then, (i) each of ground pairs of each of the estimated regions and each of their corresponding GT ground regions and (ii) each of regions and sensor reading pairs of each of the estimated regions, in order to generate at least one reflectivity total loss and at least one ice presence loss, by using any of loss generating algorithms, e.g., a smooth-L1 loss algorithm and a cross-entropy loss algorithm. Thereafter, by referring to the reflectivity total loss and black ice presence loss, backpropagation may have been performed to learn at least part of parameters of the data extraction network2000. Parameters of the RPN can be trained also, but a usage of the RPN is a well-known prior art, thus further explanation is omitted.

Herein, the data vectorizing layer240may have been implemented by using a rule-based algorithm, not a neural network algorithm. In this case, the data vectorizing layer240may not need to be trained, and may just be able to perform properly by using its settings inputted by a manager.

As an example, the first feature extracting layer210, the ROI pooling layer220and the first outputting layer230may be acquired by applying a transfer learning, which is a well-known prior art, to an existing object detection network such as VGG or ResNet, etc.

Second, by referring toFIG.13, the data analysis network3000may have been trained by using (i) a plurality of source vectors for training, including apparent black ice conditions and sensor readings for training as their components, and (ii) a plurality of their corresponding GT ice detection confidences. More specifically, the data analysis network3000may have applied aforementioned operations to the source vectors for training, to thereby calculate their corresponding estimated total ice thickness levels for training. Then each of roadway region and ice detection levels pairs of each of the estimated regions and each of their corresponding GT regions may have been referred to, in order to generate at least one black ice detection, by using said any of loss algorithms. Thereafter, by referring to the ice detection loss, backpropagation can be performed to learn at least part of parameters of the data analysis network3000.

After performing such training processes, the computing device1000can properly calculate the estimated ice presence detection level by using the subject image including the scene photographed from the IDS member and from sensor levels.

Obviously, numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the disclosure may be practiced otherwise than as specifically described herein.

List of Elements1Black Ice2Pole3Roadway, Pathway4Traffic Light Pole5Roadway Light Pole6Sidewalk7Intersection8Audio Device9Signaling Device10IDS Member11Zone Section12Crosswalk13Light Source15IDS Member Housing18Sensing Device60Camera, Image Sensor210First FE Layer212, 950Transceiver220ROI Pooling Layer230First Output Layer240Data Vectorization Layer263Radio Front End264Battery265Input/Output Interface266Location Circuitry267Interface279Timer500, 870Display507, 509Display panel678Memory Card800, 805, 835, 1000Computer, Processor Circuitry810Network815Remote Computer820, 825, 830Server837Bus840, 845Memory848Program, Code850, 855, 865Interface860External Devices930Power Supply2000Data Extraction Network3000Data Analysis Network3100Second FE Layer3200Second Outputting Layer