IN-VEHICLE DEVICE, INFORMATION PROCESSING DEVICE, SENSOR DATA TRANSMISSION METHOD, AND INFORMATION PROCESSING METHOD

An in-vehicle device according to the present disclosure includes a processor, and a memory. The memory having instructions that, when executed by the processor, cause the processor to perform operations: acquiring vehicle state information indicating a state of a vehicle, and sensor data measured by a sensor provided on the vehicle, compressing the sensor data at a compression rate varying depending on the vehicle state information, and transmitting the compression rate in association with the compressed sensor data. The vehicle state information includes at least vehicle speed of the vehicle. The compression rate is a value determined according to whether the vehicle speed is higher than a first threshold.

FIELD

The present disclosure relates to an in-vehicle device, an information processing device, a sensor data transmission method, and an information processing method.

BACKGROUND

There has been known a technique for transmitting sensor data measured by various sensors provided on a vehicle from an in-vehicle device to a cloud server or an edge server.

A related technique is described in JP 2020-107291 A.

The present disclosure provides an in-vehicle device, an information processing device, a sensor data transmission method, and an information processing method that enable execution of processing depending on a compression rate of sensor data.

SUMMARY

An in-vehicle device according to the present disclosure includes a processor, and a memory. The memory having instructions that, when executed by the processor, cause the processor to perform operations comprising: acquiring vehicle state information indicating a state of a vehicle, and sensor data measured by a sensor provided on the vehicle, compressing the sensor data at a compression rate varying depending on the vehicle state information, and transmitting the compression rate in association with the compressed sensor data. The vehicle state information includes at least vehicle speed of the vehicle. The compression rate is a value determined according to whether the vehicle speed is higher than a first threshold.

DETAILED DESCRIPTION

Embodiments of an in-vehicle device and an information processing device according to the present disclosure are explained below with reference to the drawings.

First Embodiment

FIG.1is a diagram illustrating an example of information communication between a vehicle1and a cloud server2according to the first embodiment. The vehicle1and the cloud server2are connected via a network such as the Internet. As illustrated inFIG.1, the vehicle1in the present embodiment transmits metadata such as a compression rate, a frame rate, and resolution together with sensor data to the cloud server2.

The cloud server2is provided in a cloud environment. The cloud server2is an example of an information processing device in the present embodiment.

The cloud server2executes processing such as ADAS (Advanced Driver-Assistance Systems) based on the sensor data according to the metadata of the sensor data transmitted from the vehicle1. For example, the cloud server2changes a threshold used for processing, a processing algorithm, or a learning model used for learning the sensor data according to the compression rate of the sensor data.

The ADAS processing executed by the cloud server2is, for example, ACC (Adaptive Cruise Control System: adaptive cruise control), FCW (Forward Collision Warning), AEBS (Advanced Emergency Braking System: collision damage reduction braking control device), PD (Pedestrian Detection), TSR (Traffic Sign Recognition), LDW (Lane Departure Warning), free space detection, automatic parking, and automated valet parking (AVP), but is not limited thereto.

The cloud server2transmits vehicle control data for controlling traveling of the vehicle1to the vehicle1based on, for example, the sensor data received from the vehicle1and the metadata of the sensor data. Besides the vehicle control data for controlling traveling of the vehicle1, the cloud server2may perform information provision or the like to a driver of the vehicle1.

The sensor data in the present embodiment includes at least image data obtained by an in-vehicle camera mounted on the vehicle1imaging the surroundings of the vehicle1. The frame rate included in the metadata of the sensor data is a frame rate of the image data and the resolution is resolution of the image data. Note that the sensor data may include measurement data by various sensors besides the image data. For example, the sensor data may include distance measurement data of the distance between an obstacle around the vehicle1and the vehicle1by a sonar or a radar mounted on the vehicle1.

Note that, when “obstacle” is referred to in the present embodiment, “obstacle” includes an object, a building, a person such as a pedestrian, a bicycle, and other vehicles. Although the cloud server2is illustrated inFIG.1, an edge server may be used instead of the cloud server2or in addition to the cloud server2.

Next, a configuration of the vehicle1in the present embodiment is explained.

FIG.2is a diagram illustrating an example of the vehicle1including an in-vehicle device100according to the first embodiment. As illustrated inFIG.2, the vehicle1includes a vehicle body12and two pairs of wheels13disposed in a predetermined direction in the vehicle body12. The two pairs of wheels13include a pair of front tires13fand a pair of rear tires13r.

Although the vehicle1illustrated inFIG.2includes four wheels13, the number of wheels13is not limited thereto. For example, the vehicle1may be a motorcycle.

The vehicle body12is coupled to the wheels13and is movable by the wheels13. In this case, the predetermined direction in which the two pairs of wheels13are disposed is a traveling direction of the vehicle1. The vehicle1can move forward or backward by switching not-illustrated gears or the like. The vehicle1can also turn to the right and the left according to steering.

The vehicle body12has a front end portion F, which is an end portion on the front tire13fside, and a rear end portion R, which is an end portion on the rear tire13rside. The vehicle body12has a substantially rectangular shape in top view. Four corners of the substantially rectangular shape are sometimes referred to as end portions. Although not illustrated inFIG.2, the vehicle1includes a display device, a speaker, and an operation unit.

A pair of bumpers14is provided at the front and rear end portions F and R of the vehicle body12and near the lower end of the vehicle body12. Of the pair of bumpers14, a front bumper14fcovers an entire front surface and a part of a side surface near the lower end portion of the vehicle body12. Of the pair of bumpers14, a rear bumper14rcovers an entire rear surface and a part of a side surface near the lower end portion of the vehicle body12.

Wave transmitting and receiving units15fand15rthat transmit and receive sound waves such as ultrasonic waves are disposed at a predetermined end of the vehicle body12. For example, one or more wave transmitting and receiving units15fare disposed in the front bumper14fand one or more wave transmitting and receiving units15rare disposed in the rear bumper14r. In the following explanation, when not being particularly limited, the wave transmitting and receiving units15fand15rare simply referred to as wave transmitting and receiving units15. The number and positions of the wave transmitting and receiving units15are not limited to the example illustrated inFIG.2. For example, the vehicle1may include the wave transmitting and receiving units15on the left and right sides.

In the present embodiment, sonars using ultrasonic waves are an example of the wave transmitting and receiving units15. However, the wave transmitting and receiving units15may be radars that transmit and receive electromagnetic waves. Alternatively, the vehicle1may include both of the sonars and the radars. The wave transmitting and receiving units15may be simply referred to as sensors.

The wave transmitting and receiving units15detect an obstacle around the vehicle1based on a result of transmission and reception of a sound wave or an electromagnetic wave. The wave transmitting and receiving units15measure the distance between the obstacle around the vehicle1and the vehicle1based on the result of transmission and reception of the sound wave or the electromagnetic wave.

The vehicle1includes a first in-vehicle camera16athat images the front of the vehicle1, a second in-vehicle camera16bthat images the rear of the vehicle1, a third in-vehicle camera16cthat images the left side of the vehicle1, and a fourth in-vehicle camera that images the right side of the vehicle1. The fourth in-vehicle camera is not illustrated.

In the following explanation, when not being particularly distinguished, the first in-vehicle camera16a, the second in-vehicle camera16b, the third in-vehicle camera16c, and the fourth in-vehicle camera are simply referred to as in-vehicle cameras16. The positions and the number of the in-vehicle cameras are not limited to the example illustrated inFIG.2. For example, the vehicle1may include two in-vehicle cameras, i.e., the first in-vehicle camera16aand the second in-vehicle camera16b. Alternatively, the vehicle1may further include another in-vehicle camera besides the example explained above.

The in-vehicle cameras16are capable of capturing videos around the vehicle1, and are, for example, cameras that capture color images. Image data captured by the in-vehicle cameras16may be a moving image or a still image. The in-vehicle cameras16may be cameras built on the vehicle1, cameras of a drive recorder retrofitted to the vehicle1, or the like.

The in-vehicle device100is mounted on the vehicle1. The in-vehicle device100is an information processing device that can be mounted on the vehicle1, and is, for example, an ECU (Electronic Control Unit) or an OBU (On Board Unit) provided inside the vehicle1. Alternatively, the in-vehicle device100may be an external device installed near a dashboard of the vehicle1. Note that the in-vehicle device100may also serve as a car navigation device or the like.

Next, a configuration in the vicinity of the driver's seat of the vehicle1in the present embodiment is explained.FIG.3is a diagram illustrating an example of a configuration in the vicinity of a driver's seat130aof the vehicle1according to the first embodiment.

As illustrated inFIG.3, the vehicle1includes the driver's seat130aand a passenger seat130b. A windshield180, a dashboard190, a steering wheel140, a display device120, and an operation button141are provided in front of the driver's seat130a.

The steering wheel140is provided in front of the driver's seat130aand can be operated by a driver. A rotation angle of the steering wheel140, in other words, a steering angle is electrically or mechanically interlocked with a change in the direction of the front tires13f, which are steering wheels. Note that the steering wheels may be the rear tires13ror may be both of the front tires13fand the rear tires13r.

The operation button141is a button capable of receiving operation by a user. Note that, in the present embodiment, the user is, for example, the driver of the vehicle1. Note that the position of the operation button141is not limited to the example illustrated inFIG.3and may be provided, for example, on the steering wheel140. The operation button141is an example of an operation unit in the present embodiment. When the display device120also serves as a touch panel, the display device120may be an example of the operation unit. An operation terminal capable of transmitting a signal from the outside of the vehicle1such as a tablet terminal, a smartphone, a remote controller, or an electronic key, which is not illustrate, to the vehicle1may be an example of the operation unit.

Next, a hardware configuration of the in-vehicle device100in the present embodiment is explained.

FIG.4is a diagram illustrating an example of a hardware configuration of the in-vehicle device100according to the first embodiment. As illustrated inFIG.4, the in-vehicle device100has a hardware configuration in which a CPU (Central Processing Unit)11A, a ROM11B, a RAM11C, a device I/F (interface)11D, a CAN (Controller Area Network) I/F11E, a NW (NetWork) I/F11F, a HDD11G, and the like are mutually connected by a bus11H and a normal computer is used.

The CPU11A is an arithmetic device that controls the entire in-vehicle device100. Note that the CPU11A is an example of a processor in the in-vehicle device100in the present embodiment and another processor or a processing circuit may be provided instead of the CPU11A.

The ROM11B, the RAM11C, and the HDD11G function as storage units. For example, the ROM11B stores a program or the like that implements various kinds of processing by the CPU11A. The RAM11C is, for example, a main storage device of the in-vehicle device100and stores data necessary for the various kinds of processing by the CPU11A.

The device I/F11D is an interface connectable to various devices. For example, the device I/F11D is connected to a GPS device11land acquires position information indicating the current position of the vehicle1from the GPS device11l. The position information is, for example, values of latitude and longitude indicating the absolute position of the vehicle1.

The GPS device11lis a device that specifies GPS coordinates representing the position of the vehicle1based on a GPS signal received by a GPS antenna11J. The GPS antenna11J is an antenna capable of receiving a GPS signal.

The device I/F11D acquires images, detection results, and the like from in-vehicle cameras16and the wave transmitting and receiving units15.

The CAN I/F11E is an interface for transmitting and receiving information to and from another ECU mounted on the vehicle1via a CAN in the vehicle1. Note that a communication standard other than the CAN may be adopted.

The NW I/F11F is a communication device capable of communicating with an external information processing device such as the cloud server2via a network such as the Internet. The NW I/F11F can perform communication by, for example, a public line such as LTE (Long Term Evolution) (registered trademark) or near field communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). In addition, the NW I/F11F of the in-vehicle device100may directly communicate with the cloud server2via the Internet or may indirectly communicate with the cloud server2via equipment such as another information processing device.

Next, functions of the in-vehicle device100in the present embodiment is explained.

FIG.5is a block diagram illustrating an example of the functions of the in-vehicle device100according to the first embodiment. As illustrated inFIG.5, the in-vehicle device100includes an acquisition unit101, a determination unit102, a generation unit103, a compression unit104, a transmission unit105, a reception unit106, a vehicle control unit107, a display control unit108, a reception unit109, and a storage unit110.

The storage unit110is configured by, for example, the ROM11B, the RAM11C, or the HDD11G. Note that, inFIG.5, one storage unit110is illustrated as being included in the in-vehicle device100. However, a plurality of storage media may function as the storage unit110.

The acquisition unit101, the determination unit102, the generation unit103, the compression unit104, the transmission unit105, the reception unit106, the vehicle control unit107, the display control unit108, and the reception unit109are functions executed by, for example, the CPU11A reading a program stored in the ROM11B or the HDD11G. Alternatively, a hardware circuit including these functions may be provided in the in-vehicle device100.

The acquisition unit101acquires vehicle state information indicating a state of the vehicle1and sensor data measured by sensors provided on the vehicle1.

In the present embodiment, the sensors provided on the vehicle1are, for example, the in-vehicle cameras16or the wave transmitting and receiving units15. More specifically, in the present embodiment, an example in which the in-vehicle cameras16are the sensors is explained. In the present embodiment, image data captured by the in-vehicle cameras16is explained as an example of sensor data.

The acquisition unit101acquires image data obtained by imaging the surroundings of the vehicle1from the in-vehicle cameras16via the device I/F11D. The acquisition unit101acquires distance measurement data of the distance between an obstacle around the vehicle1and the vehicle1and a detection result of the obstacle from the wave transmitting and receiving units15via the device I/F11D.

Note that the sensors may be both in-vehicle cameras16and the wave transmitting and receiving units15or the wave transmitting and receiving units15. The vehicle1may further include other sensors. When the sensors are the wave transmitting and receiving units15, the sensor data may be distance measurement data of the distance between an obstacle around the vehicle1and the vehicle1or detection results of the obstacle by the wave transmitting and receiving units15.

The vehicle state information in the present embodiment includes at least vehicle speed of the vehicle1. The acquisition unit101acquires the vehicle speed of the vehicle1from other ECUs via the CAN I/F11E. The acquisition unit101may acquire wheel speed of the vehicle1from a wheel speed sensor or the like provided on the vehicle1and obtain vehicle speed of the vehicle1from the wheel speed.

The vehicle state information may include information other than the vehicle speed. For example, at least one of vehicle speed, GPS position information of the vehicle1measured by the GPS device11l, a state of the gears of the vehicle1, a steering angle, peripheral information such as illuminance around the vehicle1, weather information around the vehicle1, and an operation state of wipers of the vehicle1may be included in the vehicle state information. These kinds of information are acquired from the devices of the vehicle1by the acquisition unit101. The GPS position information is used to specify characteristics of the current position of the vehicle1. For example, the acquisition unit101may obtain the characteristics of the current position of the vehicle1from map data and the GPS position information of the vehicle1. The map data may be stored in, for example, an external device connected to the in-vehicle device100via the Internet or the like or may be stored in the storage unit110of the in-vehicle device100. The characteristics of the current position of the vehicle1are, for example, classifications such as a parking lot, a general road, and a freeway.

The determination unit102determines a compression rate of sensor data according to the vehicle state information. For example, when the vehicle speed included in the vehicle state information is equal to or lower than a first threshold, the determination unit102determines to compress the sensor data at a low compression rate. When the vehicle speed included in the vehicle state information is higher than the first threshold, the determination unit102determines to compress the sensor data at a high compression rate. The low compression rate is an example of a first compression rate in this application. The high compression rate is an example of a second compression rate in the present application. Specific numerical values of the low compression rate and the high compression rate are not particularly limited, and it is enough that the high compression rate is higher than the low compression rate. A value of the first threshold is not particularly limited.

In the present embodiment, the compression rate is divided into two stages of the low compression rate and the high compression rate. However, the compression rate may be divided into three or more stages. For example, the determination unit102may determine the compression rate in stages such that the compression rate is higher as the vehicle speed is higher.

When the vehicle speed is high, the in-vehicle device100desirably quickly obtains a result of the ADAS processing from the cloud server2in order for driving control of the vehicle1. As the compression rate of the sensor data is higher, a transmission time of the compressed sensor data between the vehicle1and the cloud server2is shorter. Therefore, the determination unit102sets the compression rate of the sensor data higher when the vehicle speed is high than when the vehicle speed is low and quickly transmits the sensor data after the compression to the cloud server2.

Note that the compression system in the present embodiment is not particularly limited but is basically irreversible compression. Therefore, the compressed sensor data transmitted from the vehicle1to the cloud server2has a smaller data size than a state before the compression even after being decompressed by the cloud server2. When the cloud server2executes pedestrian detection processing based on image data transmitted from the vehicle1, as a size of the image data is larger, more highly accurate pedestrian detection is possible. For example, when the vehicle1is traveling at low speed, it is assumed that the vehicle1is in a state just before parking or is traveling on a narrow road. In such a case, there is a high possibility of, for example, a pedestrian rushing out. Therefore, the cloud server2executes pedestrian detection with high accuracy using high-resolution image data.

The determination unit102may determine the compression rate of the sensor data according to the vehicle state information other than the vehicle speed. For example, when the wipers of the vehicle1are continuously operating, raining is highly likely. When the weather is rain, the visibility of the periphery of the vehicle1by the driver decreases. In such a case, in order to improve the accuracy of pedestrian detection by the cloud server2, the determination unit102determines the compression rate of the image data to be the low compression rate. When the current position of the vehicle1is a parking lot, the determination unit102may determine the compression rate of the image data to be the low compression rate in order to improve the accuracy of detection of a pedestrian and other surrounding vehicles by the cloud server2.

The determination unit102may determine the compression rate of the sensor data according to a combination of a plurality of kinds of information, for example, a combination of the vehicle speed and the characteristics of the current position of the vehicle1.

Besides the compression rate, the determination unit102may determine various characteristics of the sensor data according to the vehicle state information. For example, when the image data is a moving image and the vehicle speed is equal to or lower than the first threshold, the determination unit102determines that the frame rate of the image data to be a low frame rate and determine the resolution of the image data to be a high resolution. When the vehicle speed is higher than the first threshold, the determination unit102determines the frame rate of the image data to be a high frame rate and the resolution of the image data to be a low resolution.

The low frame rate is an example of a first frame rate in the present embodiment. In addition, the high frame rate is an example of the first frame rate in the present embodiment. Specific values of the low frame rate and the high frame rate are not particularly limited, and it is enough that the high frame rate is higher than the low frame rate. The high resolution is an example of a first resolution in the present embodiment. The low resolution is an example of a second resolution in the present embodiment. Specific values of the high resolution and the low resolution are not particularly limited, and it is enough that the high resolution is higher than the low resolution. Note that the resolution determined by the determination unit102is a resolution in a state in which the image data is decompressed in the cloud server2after being compressed. Note that, when the image data is a still image, the determination unit102does not determine the frame rate but determines the resolution.

In the present embodiment, the frame rate and the resolution are in two stages. However, the frame rate and the resolution may be divided into three or more stages. For example, the determination unit102may determine the frame rate in stages such that the frame rate is higher as the vehicle speed is higher. In addition, the determination unit102may determine the resolution in stages such that the resolution is higher as the vehicle speed is lower. In general, movement in the image data is larger as the vehicle speed of the vehicle1is higher. The determination unit102sets the frame rate high, whereby it is possible to reduce omission of detection of the movement in the image data in the cloud server2.

The generation unit103generates header-added sensor data obtained by adding the compression rate determined by the determination unit102to the sensor data. The header-added sensor data is an example of compression-rate-added sensor data in the present embodiment.

In addition, when the determination unit102determines the frame rate and the resolution of the image data according to the vehicle state information, the generation unit103edits the image data to the frame rate determined by the determination unit102. After compressing and decompressing the image data, the generation unit103may adjust the resolution of the image data to be the resolution determined by the determination unit102.

FIG.6is a diagram illustrating an example of header-added sensor data400according to the first embodiment. As illustrated inFIG.6, the header-added sensor data400is data to which a compression rate is given as header data410accompanying sensor data420, which is a main body.

Note that the header-added sensor data400may include data other than the compression rate in the header data410. In the example illustrated inFIG.6, the header data410includes the compression rate of the image data, the frame rate of the image data, and the resolution of the image data. The compression rate of the image data, the frame rate of the image data, and the resolution of the image data are examples of metadata of the image data. When the sensor data is not image data, content of the metadata is sometimes different from that in the example illustrated inFIG.6.

Note that, in the header-added sensor data400illustrated inFIG.6, the metadata such as the compression rate is included in the header data410of the sensor data420, which is the image data. However, it is enough that the metadata such as the compression rate is associated with the sensor data420and the metadata may not necessarily be added to the sensor data420as the header data410.

Referring back toFIG.5, the compression unit104compresses the sensor data at a compression rate varying depending on the vehicle state information. More specifically, the compression unit104compresses the header-added sensor data400generated by the generation unit103at the compression rate determined by the determination unit102.

The transmission unit105transmits the compression rate to the cloud server2via the NW I/F11F in association with the compressed sensor data420. More specifically, the transmission unit105transmits the header-added sensor data400compressed by the compression unit104to the cloud server2.

The reception unit106receives vehicle control data for controlling traveling of the vehicle1from the cloud server2via the NW I/F11F. The vehicle control data is, for example, a processing result by the ADAS. Specifically, the vehicle control data is a detection result of an obstacle around the vehicle1, a signal for instructing braking operation for the vehicle1, a signal for instructing steering of the vehicle1, a signal for instructing vehicle speed of the vehicle1, or the like.

The vehicle control unit107controls traveling of the vehicle1based on the vehicle control data received by the reception unit106. For example, the vehicle control unit107controls steering, braking, and acceleration/deceleration of the vehicle1based on the vehicle control data. Besides the control based on the vehicle control data, the vehicle control unit107may control the traveling of the vehicle1based on an image of the surroundings of the vehicle1captured by the in-vehicle camera16and a distance to an obstacle around the vehicle1detected by the wave transmitting and receiving unit15.

The display control unit108causes the display device120to display various images and a GUI (Graphical User Interface). The display control unit108may cause, based on the vehicle control data received by the reception unit106, the display device120to display a warning of obstacle detection.

The reception unit109receives various kinds of operation from the driver of the vehicle1via the operation button141. When the display device120includes a touch panel, the reception unit109receives various kinds of operation input to the touch panel from the driver of the vehicle1.

Next, functions of the cloud server2are explained.

FIG.7is a block diagram illustrating an example of the functions of the cloud server2according to the first embodiment.

As illustrated inFIG.7, the cloud server2includes a reception unit201, an ADAS processing unit200, a transmission unit205, and a storage unit210. The ADAS processing unit200is an example of a control processing unit in the present embodiment.

The storage unit210is configured by, for example, a ROM, a RAM, or an HDD in a cloud environment.

The reception unit201, the ADAS processing unit200, and the transmission unit205are functions executed by, for example, a CPU in the cloud environment reading a program stored in the storage unit210.

The reception unit201receives the compressed sensor data420and the compression rate of the sensor data420from the in-vehicle device100mounted on the vehicle1. More specifically, the reception unit201receives the compressed header-added sensor data400from the in-vehicle device100mounted on the vehicle1.

The ADAS processing unit200executes processing varying depending on the compression rate of the sensor data420. More specifically, the ADAS processing unit200executes processing varying depending on the compression rate included in the compressed header-added sensor data400.

In the example illustrated inFIG.7, the ADAS processing unit200includes an obstacle detection unit202, a learning unit203, and an estimation unit204. These functional units are an example of ADAS functions executed by the ADAS processing unit200. Note that functional units included in the ADAS processing unit200are not limited thereto.

The obstacle detection unit202changes, according to the compression rate of the sensor data420, a threshold for obstacle detection in processing for detecting an obstacle from the sensor data420. The threshold for obstacle detection is hereinafter referred to as detection threshold. Note that the obstacle detection unit202decompresses the compressed sensor data420and then uses the decompressed sensor data420for obstacle detection.

For example, when the compression rate of the sensor data420is equal to or lower than a second threshold, the obstacle detection unit202executes obstacle detection based on the sensor data420using a low compression rate detection threshold. When the compression rate of the sensor data420is higher than the second threshold, the obstacle detection unit202executes obstacle detection based on the sensor data420using a high compression rate detection threshold.

The low compression rate detection threshold is an example of a first detection threshold in the present embodiment. The high compression rate detection threshold is an example of a second detection threshold in the present embodiment. The high compression rate detection threshold is a value lower than the low compression rate detection threshold. In other words, when the compression rate of the sensor data420is high, the obstacle detection unit202more easily determines that “an obstacle is present around the vehicle1” than when the compression rate of the sensor data420is low. When the sensor data420is image data, image data with a high compression rate has a lower resolution than image data with a low compression rate. Therefore, when the image data with the high compression rate is used, the obstacle detection unit202sets the detection threshold for obstacle detection lower and reduces detection omission of an obstacle. Note that a level of the detection threshold may be divided into two or more stages.

The obstacle detection unit202may generate a signal for instructing braking operation for the vehicle1for avoiding a detected obstacle, a signal for instructing steering of the vehicle1, a signal for instructing vehicle speed of the vehicle1, or the like.

The learning unit203learns sensor data transmitted from the in-vehicle device100. A method of the learning is, for example, deep learning using a learning model but is not limited thereto. Note that the learning unit203decompresses the compressed sensor data420and then uses the decompressed sensor data420for learning.

The learning unit203uses a different learning model according to the compression rate of the sensor data420. For example, when the compression rate of the sensor data420is equal to or lower than the second threshold, the learning unit203causes a low compression rate learning model to learn the sensor data420. When the compression rate of the sensor data420is higher than the second threshold, the learning unit203causes a high compression rate learning model to learn the sensor data420. The low compression rate learning model is an example of a first learning model in the present embodiment. The high compression rate learning model is an example of a second learning model in the present embodiment. The learning unit203stores the low compression rate learning model and the high compression rate learning model in, for example, the storage unit210.

Since conditions of the learning data input to the learning model are unified, learning accuracy of the learning model is improved. As explained above, when the sensor data420is image data, image data having a high compression rate has a lower resolution than image data having a low compression rate. Therefore, the learning unit203uses a different learning model according to the compression rate of the sensor data420, whereby the learning accuracy of the learning model is further improved than when the learning unit203inputs a plurality of pieces of sensor data420having different compression rates to one learning model. Note that the number of learning models is not limited to two and may be three or more.

The estimation unit204executes estimation processing using the low compression rate learning model and the high compression rate learning model learned by the learning unit203. Content of the estimation processing may be, for example, obstacle detection or forward collision warning.

For example, when the compression rate of the sensor data420is equal to or lower than the second threshold, the estimation unit204inputs the sensor data420to the learned low compression rate learning model and obtains an output from the learned low compression rate learning model as an estimation result. When the compression rate of the sensor data420is higher than the second threshold, the estimation unit204inputs the sensor data420to the learned high compression rate learning model and obtains an output from the learned high compression rate learning model as an estimation result.

Note that the learning unit203and the estimation unit204may not function simultaneously. For example, the estimation unit204may not function until the sensor data420equal to or more than a specified threshold is learned in the low compression rate learning model and the high compression rate learning model. Although the obstacle detection unit202and the estimation unit204are illustrated as separate functional units inFIG.7, the obstacle detection unit202may execute the obstacle detection using the low compression rate learning model and the high compression rate learning model learned by the learning unit203.

Note that a change of processing corresponding to the compression rate of the sensor data420is not limited to the example explained above. The ADAS processing unit200may change the processing algorithm according to the compression rate of the sensor data420. The ADAS processing unit200may execute different processing according to metadata other than the compression rate. For example, the ADAS processing unit200may change a detection threshold, a learning model, or an algorithm used for the processing according to the frame rate or the resolution included in the header data410of the header-added sensor data400.

The transmission unit205transmits vehicle control data, which is a result of the processing by the ADAS processing unit200, to the in-vehicle device100. For example, the transmission unit205transmits the detection result of the obstacle by the obstacle detection unit202to the in-vehicle device100. The transmission unit205may transmit, to the in-vehicle device100, a signal for instructing braking operation for the vehicle1, a signal for instructing steering of the vehicle1, or a signal for instructing a vehicle speed of the vehicle1, the signal being generated by the obstacle detection unit202.

Next, a flow of transmission processing for the sensor data420executed by the in-vehicle device100configured as explained above is explained.

FIG.8is a flowchart illustrating an example of a flow of the transmission processing for the sensor data420according to the first embodiment. For example, when the in-vehicle device100receives power supply from an ignition power supply of the vehicle1, the processing of this flowchart starts when the ignition power supply is in an on state. When the in-vehicle device100receives power supply from an accessory power supply of the vehicle1, the processing of this flowchart starts when the accessory power supply is in the on state.

First, the acquisition unit101acquires vehicle state information such as vehicle speed of the vehicle1(S101). The acquisition unit101acquires the sensor data420(S102).

Next, the determination unit102determines whether the vehicle1is traveling (S103). For example, when the vehicle speed of the vehicle1acquired by the acquisition unit101is equal to or higher than a threshold, the determination unit102determines that the vehicle1is traveling. Note that the determination unit102may determine presence or absence of traveling of the vehicle1based on, for example, an operation state of an accelerator pedal.

When the determination unit102determines that the vehicle1is not traveling (S103“No”), the processing returns to S101. When the vehicle1does not start traveling, the processing in S101to S103is repeatedly executed. When the vehicle1is not traveling, the acquisition unit101may not acquire the sensor data420.

When determining that the vehicle1is traveling (S103“Yes”), the determination unit102determines a compression rate, a frame rate, and a resolution of the sensor data420according to the vehicle state information (S104).

Subsequently, the generation unit103edits the sensor data420according to the frame rate and the resolution determined by the determination unit102(S105).

Subsequently, the generation unit103describes the compression rate, the frame rate, and the resolution determined by the determination unit102in a header of the sensor data420(S106).

Next, the compression unit104compresses the header-added sensor data400at the compression rate determined by the determination unit102(S107).

Next, the transmission unit105transmits the compressed header-added sensor data400to the cloud server2(S108).

When the ignition power supply or the accessory power supply of the vehicle1is in the on state (S109“No”), the processing returns to S101and the processing of this flowchart is repeated. When the ignition power supply or the accessory power supply of the vehicle1is turned off (S109“Yes”), the processing of this flowchart ends.

Next, a flow of ADAS processing executed by the cloud server2is explained.

FIG.9is a flowchart illustrating an example of a flow of ADAS processing executed by the cloud server2according to the first embodiment. The ADAS processing by the cloud server2is basically always in a start possible state and is on standby for reception of the header-added sensor data400from the in-vehicle device100.

When the reception unit201of the cloud server2receives the header-added sensor data400from the in-vehicle device100mounted on the vehicle1(S201“Yes”) and the compression rate is the second threshold or lower (S202“Yes”), the obstacle detection unit202, the learning unit203, and the estimation unit204included in the ADAS processing unit200of the cloud server2execute various kinds of ADAS processing by applying the low compression rate detection threshold, the low compression rate learning model, and a low compression rate algorithm (S203).

When the reception unit201receives the header-added sensor data400from the in-vehicle device100mounted on the vehicle1(S201“Yes”) and the compression rate is higher than the second threshold (S202“No”), the obstacle detection unit202, the learning unit203, and the estimation unit204included in the ADAS processing unit200of the cloud server2execute the ADAS processing by applying the high compression rate detection threshold, the high compression rate learning model, and a high compression rate algorithm (S204).

After the processing in S203and S204, the transmission unit205of the cloud server2transmits vehicle control data, which is a result of the processing by the ADAS processing unit200, to the in-vehicle device100(S205).

Then, when the ADAS processing is continued without ending (S206“No”), the processing returns to S201and the processing of this flowchart is repeated. When the ADAS processing ends (S206“Yes”), the processing of this flowchart ends. The ADAS processing by the cloud server2ends, for example, when the cloud server2is stopped by an administrator.

As explained above, the in-vehicle device100in the present embodiment compresses the sensor data420at a compression rate varying depending on the vehicle state information of the vehicle1and transmits the compression rate in association with the compressed sensor data420. Therefore, the cloud server2or the edge server that has received sensor data after compression from the in-vehicle device100in the present embodiment is capable of executing processing corresponding to a compression rate of the sensor data.

As a comparative example, when sensor data is compressed at a compression rate varying depending on vehicle state information and the compressed sensor data is solely transmitted to a cloud server or an edge server, on the side of the cloud server, the edge server, or the like that has received the compressed sensor data, the compression rate of the received compressed sensor data cannot be specified. Therefore, it is difficult to change processing according to the compression rate. Therefore, in the comparative example, the same processing is applied to sensor data having various data sizes and the like because compression rates are different. On the other hand, since the in-vehicle device100in the present embodiment transmits the compression rate in association with the compressed sensor data420, the cloud server2or the edge server that has received the compressed sensor data420is capable of dividing the processing according to the compression rate and can improve the accuracy of the processing result.

The in-vehicle device100in the present embodiment generates the header-added sensor data400obtained by adding the compression rate determined according to the vehicle state information as the header data to the acquired sensor data, compresses the header-added sensor data400at the determined compression rate, and transmits the compressed header-added sensor data400. Therefore, with the in-vehicle device100in the present embodiment, since the compression rate and the sensor data420can be integrally transmitted to the cloud server2or the like, it is easy to grasp a correspondence relation between the compression rate and the sensor data420on the reception side.

In the present embodiment, the vehicle state information includes at least the vehicle speed of the vehicle1. The in-vehicle device100in the present embodiment determines to compress the sensor data420at the low compression rate when the vehicle speed is equal to or lower than the first threshold and determines to compress the sensor data420at the high compression rate when the vehicle speed is higher than the first threshold. Therefore, with the in-vehicle device100in the present embodiment, when the vehicle speed of the vehicle1is high, by increasing the compression rate of the sensor data420to shorten a data transmission time to the cloud server2or the like, it is possible to quickly obtain the processing result from the cloud server2or the like. In addition, with the in-vehicle device100in the present embodiment, when the vehicle speed of the vehicle1is low, it is possible to suppress the compression rate of the sensor data420to reduce deterioration of the sensor data420and improve the accuracy of the ADAS processing in the cloud server2or the like. In other words, the in-vehicle device100in the present embodiment can appropriately adjust the balance between the data transmission speed and the accuracy of the ADAS processing according to the vehicle speed of the vehicle1.

In the present embodiment, the sensor data420includes at least the image data obtained by imaging the surroundings of the vehicle1. When the vehicle speed of the vehicle1is equal to or lower than the first threshold, the in-vehicle device100in the present embodiment determines the frame rate of the image data to be the low frame rate and the resolution of the image data to be the high resolution. When the vehicle speed of the vehicle1is higher than the first threshold, the in-vehicle device100in the present embodiment determines the frame rate of the image data to be the high frame rate and the resolution of the image data to be the low resolution. Therefore, with the in-vehicle device100in the present embodiment, in addition to the adjustment of the balance between the data transmission speed and the accuracy of the ADAS processing by the compression rate, further, it is possible to adjust the balance between the data transmission speed and the accuracy of the ADAS processing corresponding to the vehicle speed of the vehicle1by the frame rate and the resolution.

The vehicle state information in the present embodiment includes at least one of the vehicle speed of the vehicle1, the position of the vehicle1, the state of the gears of the vehicle1, the steering angle of the vehicle1, the illuminance around the vehicle1, the weather information around the vehicle1, and the operation state of the wipers of the vehicle1. Therefore, with the in-vehicle device100in the present embodiment, it is possible to appropriately adjust the balance between the data transmission speed and the accuracy of the ADAS processing according to not only the vehicle speed but also various states of the vehicle1.

The cloud server2in the present embodiment receives the compressed sensor data420and the compression rate of the sensor data420from the in-vehicle device100and executes processing varying depending on the compression rate of the sensor data420. Therefore, with the cloud server2in the present embodiment, it is possible to divide the processing according to the compression rate and it is possible to improve the accuracy of the processing result.

In addition, the cloud server2in the present embodiment executes the obstacle detection based on the sensor data420using the low compression rate detection threshold when the compression rate of the sensor data420is equal to or lower than the second threshold and executes the obstacle detection based on the sensor data420using the high compression rate detection threshold when the compression rate of the sensor data420is higher than the second threshold. Therefore, with the cloud server2in the present embodiment, when the image data with the high compression rate is used, it is possible to reduce detection omission of an obstacle by reducing the detection threshold for obstacle detection.

The cloud server2in the present embodiment causes the low compression rate learning model to learn the sensor data420when the compression rate is equal to or lower than the second threshold and causes the high compression rate learning model to learn the sensor data420when the compression rate is higher than the second threshold. Therefore, with the cloud server2in the present embodiment, by using a different learning model according to the compression rate, it is possible to further improve learning accuracy of a learning model than when a plurality of pieces of sensor data420having different compression rates are input to one learning model.

Second Embodiment

In the first embodiment explained above, the cloud server2executes the ADAS processing based on the sensor data420transmitted from the in-vehicle device100. However, an execution subject of the ADAS processing is not limited to the cloud server2. For example, an edge server may be the execution subject of the ADAS processing.

FIG.10is a diagram illustrating an example of information communication among the vehicle1, an edge server3, and the cloud server2according to a second embodiment. In the present embodiment, the edge server3and the cloud server2are examples of an information processing device.

The edge server3is a computer capable of storing processing of the in-vehicle device100with an edge computing technology. The edge server3includes, for example, a processor such as a CPU and storage devices such as a RAM, a ROM, and an HDD. The edge server3is provided, for example, in a communication base station, a traffic infrastructure facility, or the like and mutually performs information communication with the in-vehicle device100mounted on the vehicle1and the cloud server2.

For example, in the present embodiment, the obstacle detection unit202and the estimation unit204in the ADAS processing unit200of the cloud server2in the first embodiment illustrated inFIG.7are included in the edge server3. The learning unit203in the ADAS processing units200of the cloud server2in the first embodiment illustrated inFIG.7is included in the cloud server2. Both of the edge server3and the cloud server2include functional units corresponding to the reception unit201, the transmission unit205, and the storage unit210of the cloud server2in the first embodiment illustrated inFIG.7.

A reception unit of the edge server3receives the compressed header-added sensor data400from the in-vehicle device100. The obstacle detection unit202and the estimation unit204of the edge server3execute processing corresponding to the compression rate like the obstacle detection unit202and the estimation unit204of the cloud server2in the first embodiment. A transmission unit of the edge server3transmits the vehicle control data based on the processing result by the obstacle detection unit202or the estimation unit204to the vehicle1. The transmitting unit of the edge server3transmits the compressed header-added sensor data400received from the in-vehicle device100to the cloud server2.

The reception unit of the cloud server2receives the compressed header-added sensor data400from the edge server3. As in the first embodiment, the learning unit203of the cloud server2executes learning with a learning model corresponding to a compression rate of the header-added sensor data400. In addition, the transmission unit of the cloud server2transmits a learned model learned by the learning unit203to the edge server3. The learned model is used, for example, by the estimation unit204or the obstacle detection unit202of the edge server3.

Since the edge server3includes the obstacle detection unit202and the learning unit203in this manner, the vehicle control data that is the processing result based on the header-added sensor data400can be transmitted to the in-vehicle device100more quickly than processing by the cloud server2.

Since limitation on a storage capacity is generally smaller in the cloud server2than in the edge server3, a large amount of the sensor data420can be learned by the cloud server2executing learning processing that takes time but does not affect driving control of the vehicle1.

Note that, inFIG.10, a case in which both of the edge server3and the cloud server2are used illustrated. However, alternatively, a configuration in which the cloud server2is absent and processing is performed by the edge server3and the in-vehicle device100may be adopted. In this case, the edge server3is an example of the information processing device.

Third Embodiment

In the second embodiment explained above, the edge server3is provided outside the vehicle1. However, the edge server3may be provided in the vehicle1.FIG.11is a diagram illustrating an example of information communication among the vehicle1, the edge server3, and the cloud server2according to a third embodiment.

As illustrated inFIG.11, the edge server3in the present embodiment is mounted on the vehicle1like the in-vehicle device100. The edge server3is mounted on the vehicle1as explained above, whereby the in-vehicle device100and the edge server3can be connected by wire in the vehicle1and improvement in communication speed is expected.

The functions of the in-vehicle device100, the cloud server2, and the edge server3in the embodiments explained above are implemented, for example, by the CPU executing programs. The programs executed by the in-vehicle device100, the cloud server2, and the edge server3in the embodiments explained above is provided by being recorded in a computer-readable recording medium such as a CD-ROM, a CD-R, a DVD (Digital Versatile Disk), or a flash memory as a file in an installable format or an executable format.

The programs executed by the in-vehicle device100, the cloud server2, and the edge server3in the embodiments explained above may be stored in a computer connected to a network such as the Internet and provided by being downloaded via the network. The programs executed by the in-vehicle device100, the cloud server2, and the edge server3in the embodiments explained above may be provided or distributed via a network such as the Internet.

The programs executed by the in-vehicle device100, the cloud server2, and the edge server3in the embodiments explained above may be provided by being incorporated in a ROM or the like in advance.