Patent Description:
Conventionally, controlling a light distribution pattern of a headlamp apparatus based on a detection result of a road user in front of a vehicle to prevent the road user from being dazzled is proposed (for example, refer to PTL <NUM>).

In a visible image obtained by photographing surroundings of a vehicle by a visible light camera in a dark environment such as night-time, noise or blur increases in an area where illuminance of irradiated light from the vehicle is low or saturation of pixels occur in an area where the illuminance of the irradiated light is high. As a result, accuracy of image recognition processing with respect to the visible image declines and, for example, detection accuracy of objects around the vehicle declines.

In this regard, PTL <NUM> fails to consider controlling a light distribution pattern of a headlamp apparatus for the purpose of improving accuracy of image recognition processing. PTL <NUM> relates to a method for pattern detection and pattern classification for tracking objects, involves obtaining multiple patterns to be detected, illuminating a scene, capturing image frame of the scene, and synchronizing exposure with reflections.

The present technique has been devised in consideration of the circumstances described above and an object thereof is to improve accuracy of image recognition processing.

An information processing apparatus according to a first aspect of the present technique is presented as claimed in claim <NUM>.

An information processing method according to a second aspect of the present technique presented as claimed in claim <NUM>.

A photographing apparatus is presented, the photographing apparatus including: an imaging element; a recognizing portion configured to perform image recognition processing with respect to an image photographed by the imaging element and calculate, for each recognized area being an area based on a recognition result of an object, a reliability of the recognition result; and an irradiation control portion configured to control an irradiation pattern of visible light in a photographing direction of the imaging element so that the reliability with respect to at least a part of the recognized areas increases.

Alighting apparatus is presented, the lighting apparatus including: a light source portion; a recognizing portion configured to perform image recognition processing with respect to an image photographed by a photographing portion and calculate, for each recognized area being an area based on a recognition result of an object, a reliability of the recognition result; and an irradiation control portion configured to control an irradiation pattern of visible light due to the light source portion in a photographing direction of the photographing portion so that the reliability with respect to at least a part of the recognized areas increases.

A mobile body is presented, the mobile body including: a photographing portion; a lighting portion configured to irradiate visible light in a photographing direction of the photographing portion; a recognizing portion configured to perform image recognition processing with respect to an image photographed by the photographing portion and calculate, for each recognized area being an area based on a recognition result of an object, a reliability of the recognition result; and an irradiation control portion configured to control an irradiation pattern of visible light of the lighting portion so that the reliability with respect to at least a part of the recognized areas increases.

In the first and second aspect of the present technique, image recognition processing is performed, for each recognized area being an area based on a recognition result of an object, a reliability of the recognition result is calculated, and an irradiation pattern of visible light in a photographing direction is controlled so that the reliability with respect to at least a part of the recognized areas increases.

According to the first and second aspect of the present technique, accuracy of image recognition processing can be improved.

It should be noted that the advantageous effects described above are not necessarily restrictive and any of the advantageous effects described in the present disclosure may apply.

Hereinafter, modes for implementing the present technique will be described. The descriptions will be given in the following order.

<FIG> is a block diagram showing a schematic configuration example of a vehicle <NUM> to which the present technique has been applied.

The vehicle <NUM> includes a vehicle control system <NUM>.

The vehicle control system <NUM> includes a plurality of control units that are connected via a communication network <NUM>. In the example shown in <FIG>, the vehicle control system <NUM> includes a drive system control unit <NUM>, a body system control unit <NUM>, a battery control unit <NUM>, an external vehicle information detecting unit <NUM>, an internal vehicle information detecting unit <NUM>, and an integrated control unit <NUM>. The communication network <NUM> that connects the plurality of control units may be a vehicle-mounted communication network compliant with an arbitrary standard such as a CAN (Controller Area Network), a LIN (Local Interconnect Network), a LAN (Local Area Network), or FlexRay (registered trademark).

Each control unit includes a microcomputer that performs arithmetic processing in accordance with various programs, a storage portion that stores programs to be executed by the microcomputer, parameters to be used in various calculations, and the like, and a drive circuit that drives various apparatuses which are control targets. Each control unit includes a network I/F for communicating with other control units via the communication network <NUM> and a communication I/F for communicating with apparatuses, sensors, and the like inside and outside the vehicle via wired communication or wireless communication. <FIG> illustrates, as functional components of the integrated control unit <NUM>, a microcomputer <NUM>, a general-purpose communication I/F <NUM>, a dedicated communication I/F <NUM>, a positioning portion <NUM>, a beacon receiving portion <NUM>, an on-board device I/F <NUM>, an audio/video output portion <NUM>, a vehicle-mounted network I/F <NUM>, and a storage portion <NUM>. The other control units similarly include a microcomputer, a communication I/F, a storage portion, and the like.

The drive system control unit <NUM> controls operations of apparatuses related to a drive system of the vehicle <NUM> in accordance with various programs. For example, the drive system control unit <NUM> functions as a control apparatus of a drive force generation apparatus for generating a drive force of the vehicle <NUM> such as an internal engine or a drive motor, a control apparatus of a drive force transmission mechanism for transmitting the drive force to wheels, a control apparatus of a steering mechanism for adjusting a steering angle of the vehicle <NUM>, a control apparatus of a braking apparatus that generates a brake force of the vehicle <NUM>, and the like. The drive system control unit <NUM> may have functions as a control apparatus of an ABS (Antilock Brake System), a control apparatus of ESC (Electronic Stability Control), or the like.

A vehicle state detecting portion <NUM> is connected to the drive system control unit <NUM>. For example, the vehicle state detecting portion <NUM> includes at least one of a gyroscope sensor that detects an angular velocity of a rotational motion of a shaft of a vehicle body, an acceleration sensor that detects an acceleration of the vehicle <NUM>, and a sensor for detecting an operation amount of a gas pedal, an operation amount of a brake pedal, a steering angle of a steering wheel, the number of revolutions of an engine, a rotational speed of a wheel, or the like. The drive system control unit <NUM> performs arithmetic processing using a signal input from the vehicle state detecting portion <NUM> and controls an internal engine, a drive motor, an electric power steering apparatus, a brake apparatus, or the like.

The body system control unit <NUM> controls operations of various apparatuses mounted to the vehicle body in accordance with various programs. For example, the body system control unit <NUM> functions as a control apparatus of a keyless entry system, a smart key system, a power window apparatus, or various lamps such as head lamps, tail lamps, brake lamps, turn indicators, and fog lamps. In this case, radio waves or signals of various switches which are transmitted from a portable device that substitutes as a key may be input to the body system control unit <NUM>. The body system control unit <NUM> accepts input of the radio waves or signals and controls a door lock apparatus, the power window apparatus, the lamps, and the like of the vehicle <NUM>.

It should be noted that <FIG> only illustrates a lighting portion <NUM> as a control target of the body system control unit <NUM>. The lighting portion <NUM> includes at least a part of the various lamps described above.

The battery control unit <NUM> controls a secondary battery (not illustrated) that is a power supply source of the drive motor in accordance with various programs. For example, information on a battery temperature, a battery output voltage, a battery remaining capacity, or the like is input to the battery control unit <NUM> from a battery apparatus including the secondary battery. The battery control unit <NUM> uses these signals to perform arithmetic processing to control temperature regulation of the secondary battery or to control a cooling apparatus or the like included in the battery apparatus.

The external vehicle information detecting unit <NUM> detects information on an exterior of the vehicle <NUM>. For example, at least one of a photographing portion <NUM> and an external vehicle information detecting portion <NUM> is connected to the external vehicle information detecting unit <NUM>. The photographing portion <NUM> includes at least one of a ToF (Time of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. For example, the external vehicle information detecting portion <NUM> includes at least one of an environmental sensor for detecting present weather or meteorological phenomena and an ambient information detection sensor for detecting other vehicles, obstacles, pedestrians, or the like around the vehicle <NUM>.

For example, the environmental sensor may be at least one of a raindrop sensor that detects rainy weather, a fog sensor that detects fog, a sunshine sensor that detects a degree of sunshine, and a snow sensor that detects snowfall. The ambient information detection sensor may be at least one of an ultrasonic sensor, a radar apparatus, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) apparatus. The photographing portion <NUM> and the external vehicle information detecting portion <NUM> may be respectively included as an independent sensor or an independent apparatus or may be included as an apparatus that integrates a plurality of sensors or apparatuses.

The external vehicle information detecting unit <NUM> causes the photographing portion <NUM> to photograph an image of the exterior of the vehicle and receives photographed image data. In addition, the external vehicle information detecting unit <NUM> receives detection information from the external vehicle information detecting portion <NUM> being connected thereto. When the external vehicle information detecting portion <NUM> is an ultrasonic sensor, a radar apparatus, or a LIDAR apparatus, the external vehicle information detecting unit <NUM> causes the external vehicle information detecting portion <NUM> to transmit ultrasonic waves, electromagnetic waves, or the like and receive information on received reflected waves (hereinafter, referred to as reflected wave information). Based on the reflected wave information, the external vehicle information detecting unit <NUM> may perform object detection processing or distance detection processing with respect to people, vehicles, obstacles, signs, characters on road surfaces, and the like. Based on the reflected wave information, the external vehicle information detecting unit <NUM> may perform environmental recognition processing for recognizing rainfall, fog, road surface conditions, or the like. Based on the reflected wave information, the external vehicle information detecting unit <NUM> may calculate a distance to an object outside of the vehicle.

In addition, based on received image data, the external vehicle information detecting unit <NUM> may perform image recognition processing or distance detection processing for recognizing people, vehicles, obstacles, signs, characters on road surfaces, and the like. The external vehicle information detecting unit <NUM> may perform processing such as distortion correction or positioning with respect to the received image data and composite the image data photographed by different photographing portions <NUM> to generate a bird's-eye view image or a panoramic image. The external vehicle information detecting unit <NUM> may perform viewpoint transformation processing using image data photographed by different photographing portions <NUM>.

The internal vehicle information detecting unit <NUM> detects information on an interior of the vehicle. For example, a driver state detecting portion <NUM> that detects a state of a driver is connected to the internal vehicle information detecting unit <NUM>. The driver state detecting portion <NUM> may include a camera that photographs the driver, a biometric sensor that detects biological information of the driver, a microphone that collects sound inside the cabin, or the like. For example, the biometric sensor is provided on a seat surface, the steering wheel, or the like, and detects biological information of a passenger sitting on the seat or the driver holding the steering wheel. Based on detection information input from the driver state detecting portion <NUM>, the internal vehicle information detecting unit <NUM> may calculate a degree of fatigue or a degree of concentration of the driver or may determine whether or not the driver has fallen asleep. The internal vehicle information detecting unit <NUM> may perform processing such as noise cancellation processing with respect to a collected sound signal.

The integrated control unit <NUM> controls overall operations in the vehicle control system <NUM> in accordance with various programs. An input portion <NUM> is connected to the integrated control unit <NUM>. The input portion <NUM> is realized by an apparatus on which a passenger can perform input operations such as a touch panel, a button, a microphone, a switch, or a lever. Data obtained by subjecting sound input from the microphone to speech recognition may be input to the integrated control unit <NUM>. For example, the input portion <NUM> may be a remote-controlled apparatus using infrared light or other radio waves or an externally-connected device such as a mobile phone or a PDA (Personal Digital Assistant) that accommodates operations of the vehicle control system <NUM>. For example, the input portion <NUM> may be a camera, in which case a passenger can input information by gesturing to the camera. Alternatively, data obtained by detecting a motion of a wearable apparatus being worn by a passenger may be input. Furthermore, for example, the input portion <NUM> described above may include an input control circuit or the like which generates an input signal based on information input by a passenger or the like using the input portion <NUM> and which outputs the generated input signal to the integrated control unit <NUM>. By operating the input portion <NUM>, a passenger or the like inputs various types of data and issues instructions to perform processing operations with respect to the vehicle control system <NUM>.

The storage portion <NUM> may include a ROM (Read Only Memory) that stores various programs to be executed by the microcomputer <NUM> and a RAM (Random Access Memory) that stores various parameters, calculation results, sensor values, or the like. In addition, the storage portion <NUM> may be realized by a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like and may provide the various programs to be executed by the microcomputer <NUM>.

The general-purpose communication I/F <NUM> is a general-purpose communication I/F that mediates communication with various devices that are present in an external environment <NUM>. The general-purpose communication I/F <NUM> may implement a cellular communication protocol such as GSM (registered trademark) (Global System of Mobile communications), WiMAX, LTE (Long Term Evolution), or LTE-A (LTE-Advanced) or another wireless communication protocol such as wireless LAN (also referred to as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). For example, the general-purpose communication I/F <NUM> may connect to a device (for example, an application server or a control server) present on an external network (for example, the Internet, a cloud network, or a company-specific network) via a base station or an access point. In addition, for example, the general-purpose communication I/F <NUM> may connect to a terminal (for example, a terminal belonging to the driver or a pedestrian, a terminal of a store, or an MTC (Machine Type Communication) terminal) that is present in a vicinity of the vehicle <NUM> using P2P (Peer To Peer) technology.

For example, the various programs to be executed by the microcomputer <NUM> may be provided from the external environment <NUM>.

The dedicated communication I/F <NUM> is a communication I/F that supports a communication protocol designed to be used in the vehicle <NUM>. For example, the dedicated communication I/F <NUM> may implement a standard protocol such as WAVE (Wireless Access in Vehicle Environment) that is a combination of IEEE <NUM>. 11p constituting a lower layer and IEEE <NUM> constituting a higher layer, DSRC (Dedicated Short Range Communications), or a cellular communication protocol. Typically, the dedicated communication I/F <NUM> carries out V2X communication that is a concept including one or more of communication between vehicles (Vehicle to Vehicle communication), communication between a road and the vehicle (Vehicle to Infrastructure communication), communication between the vehicle <NUM> and a home (Vehicle to Home communication), and communication between a pedestrian and the vehicle (Vehicle to Pedestrian communication).

For example, the positioning portion <NUM> receives a GNSS (Global Navigation Satellite System) signal from a GNSS satellite (for example, a GPS (Global Positioning System) signal from a GPS satellite) and executes positioning, and generates positional information including a latitude, a longitude, and an elevation of the vehicle <NUM>. Alternatively, the positioning portion <NUM> may specify a current position by exchanging signals with a wireless access point or acquire positional information from a terminal such as a mobile phone, a PHS, or a smartphone with a positioning function.

For example, the beacon receiving portion <NUM> receives radio waves or electromagnetic waves emitted from a radio station or the like installed on a road and acquires information such as a current position, congestions, closures, and required time. Alternatively, the function of the beacon receiving portion <NUM> may be included in the dedicated communication I/F <NUM> described above.

The on-board device I/F <NUM> is a communication interface that mediates connections between the microcomputer <NUM> and various on-board devices <NUM> that are present inside the vehicle. The on-board device I/F <NUM> may establish a wireless connection using a wireless communication protocol such as a wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). In addition, the on-board device I/F <NUM> may establish, via a connection terminal (not illustrated) (and a cable when necessary), a wired connection such as USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), or MHL (Mobile High-definition Link). For example, the on-board devices <NUM> may include at least one of a mobile device or a wearable device that is held or worn by a passenger and an information device to be carried onto or attached to the vehicle <NUM>. Furthermore, the on-board devices <NUM> may include a navigation apparatus that searches a route to an arbitrary destination. The on-board device I/F <NUM> exchanges control signals and data signals with the on-board devices <NUM>.

The vehicle-mounted network I/F <NUM> is an interface that mediates communication between the microcomputer <NUM> and the communication network <NUM>. The vehicle-mounted network I/F <NUM> transmits and receives signals and the like in accordance with a prescribed protocol that is supported by the communication network <NUM>.

The microcomputer <NUM> of the integrated control unit <NUM> controls the vehicle control system <NUM> in accordance with various programs based on information acquired via at least one of the general-purpose communication I/F <NUM>, the dedicated communication I/F <NUM>, the positioning portion <NUM>, the beacon receiving portion <NUM>, the on-board device I/F <NUM>, and the vehicle-mounted network I/F <NUM>. For example, based on acquired information on the exterior and the interior of the vehicle, the microcomputer <NUM> may calculate a control target value of the drive force generation apparatus, the steering mechanism, or the brake apparatus and output a control command to the drive system control unit <NUM>. For example, the microcomputer <NUM> may perform cooperative control for the purpose of realizing functions of an ADAS (Advanced Driver Assistance System) including collision avoidance or crash mitigation of the vehicle <NUM>, headway control based on inter-vehicular distance, cruise control, a collision warning of the vehicle <NUM>, and a lane departure warning of the vehicle <NUM>. In addition, by controlling the drive force generation apparatus, the steering mechanism, the brake apparatus, or the like based on acquired information on the surroundings of the vehicle <NUM>, the microcomputer <NUM> may perform cooperative control for the purpose of automated driving or the like that enables the vehicle to travel autonomously without having to rely on operations by the driver.

The microcomputer <NUM> may generate three-dimensional distance information between the vehicle <NUM> and a surrounding object such as a structure or a person and create local map information including peripheral information of a current position of the vehicle <NUM> based on information acquired via at least one of the general-purpose communication I/F <NUM>, the dedicated communication I/F <NUM>, the positioning portion <NUM>, the beacon receiving portion <NUM>, the on-board device I/F <NUM>, and the vehicle-mounted network I/F <NUM>. In addition, based on acquired information, the microcomputer <NUM> may predict danger such as a collision involving the vehicle <NUM>, an approach by a pedestrian or the like, or entering a closed road and generate a warning signal. For example, the warning signal may be a signal for generating a warning sound or turning on a warning lamp.

The audio/video output portion <NUM> transmits an output signal of at least one of sound and an image to an output apparatus that is capable of audibly or visually notifying information to a passenger of the vehicle <NUM> or to the outside of the vehicle <NUM>. In an example shown in <FIG>, an audio speaker <NUM>, a display portion <NUM>, and an instrument panel <NUM> are exemplified as output apparatuses. For example, the display portion <NUM> may include at least one of an on-board display and a head-up display. The display portion <NUM> may have an AR (Augmented Reality) display function. The output apparatus may be an apparatus other than those described above such as headphones, a wearable device such as a spectacle-type display that is worn by a passenger, a projector, or a lamp. When the output apparatus is a display apparatus, the display apparatus displays, in various formats such as a text, an image, a table, and a graph, results obtained by various types of processing performed by the microcomputer <NUM> and information received from other control units. In addition, when the output apparatus is an audio output apparatus, the audio output apparatus converts an audio signal constituted by reproduced speech data, acoustic data, or the like into an analog signal and auditorially outputs the converted analog signal.

In the example shown in <FIG>, at least two control units connected via the communication network <NUM> may be integrated as a single control unit. Alternatively, each control unit may be constituted by a plurality of control units. Furthermore, the vehicle control system <NUM> may include other control units that are not illustrated. In addition, a part of or all of the functions assumed by any control unit in the description provided above may be shouldered by another control unit. In other words, when information is to be transmitted and received via the communication network <NUM>, prescribed arithmetic processing may be performed by any control unit. In a similar manner, a sensor or an apparatus connected to any control unit may be connected to another control unit and, at the same time, a plurality of control units may mutually transmit and receive detection information via the communication network <NUM>.

Hereinafter, a description of the communication network <NUM> when a control unit performs communication via the communication network <NUM> will be omitted. For example, when the drive system control unit <NUM> and the integrated control unit <NUM> perform communication via the communication network <NUM>, a simple description of "the drive system control unit <NUM> and the integrated control unit <NUM> perform communication" will be used.

In addition, hereinafter, a description of the vehicle-mounted network I/F <NUM> and the communication network <NUM> when the microcomputer <NUM> of the integrated control unit <NUM> performs communication via the vehicle-mounted network I/F <NUM> and the communication network <NUM> will be omitted. For example, when the microcomputer <NUM> performs communication with the drive system control unit <NUM> via the vehicle-mounted network I/F <NUM> and the communication network <NUM>, a simple description of "the microcomputer <NUM> performs communication with the drive system control unit <NUM>" will be used.

<FIG> is a block diagram showing a part of a configuration example of the lighting portion <NUM> shown in <FIG>.

The lighting portion <NUM> includes a variable irradiation lamp <NUM> and an infrared lamp <NUM>.

The variable irradiation lamp <NUM> includes a visible light source portion <NUM> that emits visible light and is installed at a position where the front of the vehicle <NUM> can be irradiated with visible light. In addition, the variable irradiation lamp <NUM> irradiates the front of the vehicle <NUM> with visible light under the control of the body system control unit <NUM>.

In addition, the variable irradiation lamp <NUM> is capable of controlling light intensity for each unit area created by dividing an irradiation area of visible light in plurality, and controls an irradiation pattern by controlling the light intensity for each unit area. Therefore, the irradiation pattern indicates a distribution of light intensity in a spatial direction of visible light emitted from the variable irradiation lamp <NUM> or, more specifically, a distribution of light intensity for each unit area of visible light.

For example, the visible light source portion <NUM> has a configuration in which a plurality of light sources (for example, LEDs) are arranged in an array. In addition, the irradiation pattern of visible light is controlled by individually controlling light intensity of each light source of the visible light source portion <NUM>. Controlling the light intensity of each light source includes setting the light intensity to <NUM> or, in other words, turning off the light source.

Alternatively, for example, the visible light source portion <NUM> includes a light source, a DMD (Digital Mirror Device), a lens, and the like. In addition, the irradiation pattern of visible light is controlled by controlling on/off states of each mirror of the DMD of the visible light source portion <NUM>.

It should be noted that all of the unit areas need not necessarily have a same shape and a same size and unit areas with a different shape or a different size may be included.

The infrared lamp <NUM> includes an infrared light source portion <NUM> that emits infrared light and is installed at a position where the front of the vehicle <NUM> can be irradiated with infrared light. In addition, the infrared lamp <NUM> irradiates the front of the vehicle <NUM> with infrared light under the control of the body system control unit <NUM>.

<FIG> is a block diagram showing a configuration example of the photographing portion <NUM> shown in <FIG>.

The photographing portion <NUM> includes a visible light camera <NUM> and an infrared camera <NUM>.

The visible light camera <NUM> includes a visible light imaging element <NUM> having sensitivity with respect to visible light and is installed at a position where the front of the vehicle <NUM> can be photographed. For example, the visible light camera <NUM> is installed inside a headlamp, on top of a dashboard, above a rear view mirror inside a cabin, on top of a roof, or the like of the vehicle <NUM>. In addition, the visible light camera <NUM> photographs the front of the vehicle <NUM> under the control of the external vehicle information detecting unit <NUM> and supplies the external vehicle information detecting unit <NUM> with data of a color image (hereinafter, referred to as a visible image) obtained as a result of the photography.

The variable irradiation lamp <NUM> described above irradiates visible light in a photographing direction of the visible light camera <NUM> (the visible light imaging element <NUM>) and is at least capable of irradiating visible light roughly evenly within an angle of view of the visible light camera <NUM>.

The infrared camera <NUM> includes an infrared light imaging element <NUM> having sensitivity with respect to infrared rays (infrared light) and is installed at a position where the front of the vehicle <NUM> can be photographed. For example, the infrared camera <NUM> is installed at a position similar to that of the visible light camera <NUM>. In addition, the infrared camera <NUM> photographs the front of the vehicle <NUM> under the control of the external vehicle information detecting unit <NUM> and supplies the external vehicle information detecting unit <NUM> with data of a black and white image (hereinafter, referred to as an infrared image) obtained as a result of the photography.

The infrared lamp <NUM> described above irradiates infrared light in a photographing direction of the infrared camera <NUM> (the infrared light imaging element <NUM>) and is at least capable of irradiating infrared light roughly evenly within an angle of view of the infrared camera <NUM>.

<FIG> shows a configuration example of the information processing portion <NUM> which is a part of functions realized as the microcomputer <NUM> of the integrated control unit <NUM> executes a prescribed control program. In the drawing, illustration of the communication network <NUM> and the vehicle-mounted network I/F <NUM> among the drive system control unit <NUM>, the body system control unit <NUM>, the external vehicle information detecting unit <NUM>, and the information processing portion <NUM> has been omitted.

The information processing portion <NUM> includes a recognizing portion <NUM>, a ranging portion <NUM>, an irradiation control portion <NUM>, and an operation control portion <NUM>.

The recognizing portion <NUM> acquires data of a visible image and an infrared image from the external vehicle information detecting unit <NUM> and performs image recognition processing with respect to the visible image and the infrared image. In this case, image recognition processing refers to, for example, processing for recognizing an object in an image. In addition, the recognizing portion <NUM> calculates, for each area (hereinafter, referred to as a recognized area) based on a recognition result of an object by the image recognition processing, a recognition score that indicates a reliability of the recognition result.

The recognized area is, for example, an area corresponding to each recognized object. However, the recognized area may also include an area in which recognition of an object has failed. In addition, for example, the recognition score indicates a reliability of a recognition result of an object in the recognized area or, in other words, a certainty of an object having been recognized in the recognized area.

The recognizing portion <NUM> supplies the irradiation control portion <NUM> and the operation control portion <NUM> with recognition information including a position of each recognized area, a type of object in each recognized area, and a recognition score.

The ranging portion <NUM> performs detection processing of a distance to an object in front of the vehicle <NUM> based on image data of a stereo camera, image data of a ToF camera, or reflected wave information of an ultrasonic sensor, a radar apparatus, or a LIDAR apparatus which is supplied from the external vehicle information detecting unit <NUM>. The ranging portion <NUM> supplies the irradiation control portion <NUM> and the operation control portion <NUM> with ranging information including a detection result of a distance to each object.

The irradiation control portion <NUM> controls irradiation of visible light and infrared light by the lighting portion <NUM> based on a recognition result of an object in front of the vehicle <NUM>, a detection result of a distance to the object in front of the vehicle <NUM>, a detection result of a speed of the vehicle <NUM> notified from the drive system control unit <NUM>, and the like.

For example, the irradiation control portion <NUM> generates a control signal (hereinafter, referred to as a visible light control signal) that includes an irradiation pattern of the variable irradiation lamp <NUM> and supplies the body system control unit <NUM> with the visible light control signal. The body system control unit <NUM> controls the irradiation pattern of the variable irradiation lamp <NUM> based on the visible light control signal.

In addition, for example, the irradiation control portion <NUM> generates a control signal (hereinafter, referred to as an infrared light control signal) that controls irradiation of infrared light by the infrared lamp <NUM> and supplies the body system control unit <NUM> with the infrared light control signal. The body system control unit <NUM> controls irradiation of infrared light by the infrared lamp <NUM> based on the infrared light control signal.

The operation control portion <NUM> controls operations of apparatuses related to a drive system of the vehicle <NUM> based on a recognition result of an object in front of the vehicle <NUM>, a detection result of a distance to the object in front of the vehicle <NUM>, and the like. For example, the operation control portion <NUM> generates a control signal (hereinafter, referred to as a drive system control signal) that includes control contents of an apparatus to be a control target and supplies the drive system control unit <NUM> with the drive system control signal. The drive system control unit <NUM> controls operations of the apparatus to be a control target based on the drive system control signal.

In addition, the operation control portion <NUM> controls operations of various apparatuses mounted to the body of the vehicle <NUM> based on the recognition result of an object in front of the vehicle <NUM>, the detection result of a distance to the object in front of the vehicle <NUM>, and the like. For example, the operation control portion <NUM> generates a control signal (hereinafter, referred to as a body system control signal) that includes control contents of an apparatus to be a control target and supplies the body system control unit <NUM> with the body system control signal. The body system control unit <NUM> controls operations of the apparatus to be a control target based on the body system control signal.

Next, lighting control processing to be executed by the vehicle <NUM> will be described with reference to the flow chart shown in <FIG>.

The processing is started when, for example, conditions to start irradiation of visible light are satisfied such as when surrounding brightness falls below a prescribed threshold during start-up of the vehicle <NUM> or when a switch of the variable irradiation lamp <NUM> is switched on. In addition, the processing is ended when, for example, conditions to end irradiation of visible light are satisfied such as when a power source of the vehicle <NUM> is turned off, when surrounding brightness of the vehicle <NUM> equals or exceeds the prescribed threshold, or when a switch of the variable irradiation lamp <NUM> is switched off.

In step S1, the vehicle control system <NUM> starts irradiation of infrared light and visible light.

Specifically, for example, the irradiation control portion <NUM> generates an infrared light control signal for control so that infrared light is irradiated at prescribed light intensity and supplies the body system control unit <NUM> with the generated infrared light control signal. The body system control unit <NUM> controls the infrared lamp <NUM> so as to irradiate infrared light at the prescribed light intensity based on the infrared light control signal. Accordingly, the front of the vehicle <NUM> is irradiated with infrared light with the prescribed light intensity.

With infrared light, since dazzle by passerby, drivers of other vehicles, and the like is not a concern, light intensity can be increased as much as possible within a range in which saturation of pixels in an infrared image does not occur.

In addition, for example, the irradiation control portion <NUM> generates a visible light control signal for control so that visible light is irradiated in a standard irradiation pattern and supplies the body system control unit <NUM> with the generated visible light control signal. The body system control unit <NUM> controls the variable irradiation lamp <NUM> so as to irradiate visible light in the standard irradiation pattern based on the visible light control signal. Accordingly, the front of the vehicle <NUM> is irradiated with visible light in a standard pattern.

In this case, a standard irradiation pattern is, for example, a pattern in which light intensity of all unit areas is set to a same initial value.

In step S2, the photographing portion <NUM> photographs an infrared image and a visible image. Specifically, the infrared camera <NUM> photographs the front of the vehicle <NUM> and supplies the recognizing portion <NUM> and the ranging portion <NUM> with data of an infrared image obtained as a result of the photography via the external vehicle information detecting unit <NUM>. In addition, the visible light camera <NUM> photographs the front of the vehicle <NUM> and supplies the recognizing portion <NUM> and the ranging portion <NUM> with data of a visible image obtained as a result of the photography via the external vehicle information detecting unit <NUM>.

In step S3, the recognizing portion <NUM> performs image recognition processing. For example, the recognizing portion <NUM> performs semantic segmentation as the image recognition processing.

Specifically, the recognizing portion <NUM> performs semantic segmentation with respect to an infrared image, performs detection of a type and a position of each object in the infrared image on a pixel level and, based on a recognition result, divides the infrared image into a plurality of recognized areas. In addition, the recognizing portion <NUM> calculates a recognition score of each recognized area in the infrared image.

Furthermore, the recognizing portion <NUM> performs semantic segmentation in a similar manner with respect to a visible image, performs detection of a type and a position of each object in the visible image on a pixel level and, based on a recognition result, divides the visible image into a plurality of recognized areas. In addition, the recognizing portion <NUM> calculates a recognition score of each recognized area in the visible image.

In this case, a recognized area is basically set for each recognized object. Therefore, each recognized area basically includes one object. However, for example, in cases where a boundary line of an object is ambiguous or the like, a plurality of objects may be included in one recognized area. In addition, an area in which recognition of an object has failed may be set as a recognized area. In such cases, an object in the recognized area is unknown.

In addition, an arbitrary method can be used as a calculation method of a recognition score.

Infrared images lack color information and contain more noise than visible images. Therefore, a result of image recognition processing (in the present example, semantic segmentation) with respect to an infrared image has lower reliability than a result of the image recognition processing with respect to a visible image.

In step S4, the vehicle control system <NUM> performs ranging processing. For example, the external vehicle information detecting portion <NUM> supplies the ranging portion <NUM> with reflected wave information via the external vehicle information detecting unit <NUM>. Based on the reflected wave information, the ranging portion <NUM> detects a distance to each object in front of the vehicle <NUM>. The ranging portion <NUM> supplies the irradiation control portion <NUM> and the operation control portion <NUM> with ranging information including a detection result of a distance to each object.

An arbitrary method can be used for the ranging processing. For example, the ranging processing can be performed using image data of a stereo camera, image data of a ToF camera, or the like.

In step S5, the drive system control unit <NUM> detects a vehicle speed. Specifically, based on a signal input from the vehicle state detecting portion <NUM>, the drive system control unit <NUM> detects a speed of the vehicle <NUM> and supplies the irradiation control portion <NUM> and the operation control portion <NUM> with information including a detection result.

In step S6, the vehicle control system <NUM> controls an irradiation pattern of visible light. Specifically, the irradiation control portion <NUM> sets the irradiation pattern by setting light intensity of each unit area. For example, the irradiation control portion <NUM> sets the irradiation pattern so as to increase a recognition score of each recognized area of the visible image as much as possible. The irradiation control portion <NUM> generates a visible light control signal that includes the set irradiation pattern and supplies the body system control unit <NUM> with the visible light control signal. The body system control unit <NUM> controls the irradiation pattern of the variable irradiation lamp <NUM> based on the visible light control signal.

An example of a setting method of an irradiation pattern will now be described.

For example, the irradiation control portion <NUM> sets the irradiation pattern based on a result of image recognition processing with respect to an infrared image.

For example, the irradiation control portion <NUM> sets light intensity of visible light with respect to a recognized area (hereinafter, referred to as a low-reliability area) of which a recognition score in the infrared image is lower than a threshold T1 so that a recognition score of the low-reliability area in a visible image increases. For example, the irradiation control portion <NUM> increases light intensity of visible light with respect to the low-reliability area (makes the low-reliability area brighter).

On the other hand, the irradiation control portion <NUM> sets light intensity of visible light with respect to a recognized area of which a recognition score in the infrared image is equal to or higher than the threshold T1 based on, for example, a type of an object in the recognized area.

For example, when a road surface is included in a recognized area, the irradiation control portion <NUM> controls the light intensity of visible light with respect to the recognized area so that the recognized area is constantly illuminated with light intensity of a certain level or higher.

For example, when a person or another vehicle is included in a recognized area, the irradiation control portion <NUM> lowers the light intensity of visible light with respect to the recognized area. Accordingly, passerby and drivers of other vehicles are prevented from becoming dazzled. Alternatively, the irradiation control portion <NUM> may lower the light intensity of visible light to <NUM> so that visible light is not irradiated in the recognized area.

For example, when an object (for example, a traffic light or a street lamp) that emits light is included in a recognized area, the irradiation control portion <NUM> lowers the light intensity of visible light with respect to the recognized area. This is because an object that emits light increases accuracy of image recognition processing increases even when not illuminated by visible light. Alternatively, the irradiation control portion <NUM> may lower the light intensity of visible light to <NUM> so that visible light is not irradiated in the recognized area.

For example, when an object (for example, a traffic sign) including a reflective member that reflects visible light is included in a recognized area, the irradiation control portion <NUM> lowers the light intensity of visible light with respect to the recognized area. This is because the recognized area becomes brighter than other areas due to the reflective member reflecting visible light and may become saturated in a visible image. The irradiation control portion <NUM> lowers the light intensity of visible light with respect to the recognized area to a level at which the recognized area does not become saturated in the visible image.

In addition, for example, the irradiation control portion <NUM> controls the irradiation pattern based on a result of image recognition processing with respect to a visible image.

For example, when a recognized area (a low-reliability area) of which a recognition score is lower than the threshold T1 in the visible image is present despite controlling the irradiation pattern based on a result of image recognition processing with respect to the infrared image, the irradiation control portion <NUM> controls light intensity of visible light with which the low-reliability area is to be irradiated based on an average value of brightness of the low-reliability area in the visible image. For example, when the average value of brightness of the low-reliability area is lower than a threshold Th2, the irradiation control portion <NUM> increases light intensity of visible light with respect to the low-reliability area (makes the low-reliability area brighter). On the other hand, when the average value of brightness of the low-reliability area is equal to or higher than the threshold Th2, the irradiation control portion <NUM> lowers the light intensity of visible light with respect to the low-reliability area (makes the low-reliability area darker).

Furthermore, for example, the irradiation control portion <NUM> sets the light intensity of visible light with respect to a recognized area based on a distance to an object in the recognized area. For example, the shorter the distance to an object in the recognized area, the further the irradiation control portion <NUM> lowers the light intensity of visible light with respect to the recognized area (makes the recognized area darker). On the other hand, the longer the distance to an object in the recognized area, the further the irradiation control portion <NUM> increases the light intensity of visible light with respect to the recognized area (makes the recognized area brighter).

In addition, for example, when a distance to an object in a recognized area is equal to or greater than a threshold Th3, the irradiation control portion <NUM> may exclude the recognized area from image recognition processing and set light intensity of irradiated light with which the recognized area is to be irradiated to <NUM>. Accordingly, for example, irradiation of a far object such as the sky with visible light is suspended.

For example, the irradiation control portion <NUM> may change the threshold Th3 in accordance with a speed of the vehicle <NUM>. For example, the irradiation control portion <NUM> increases the threshold Th3 as the speed of the vehicle <NUM> increases so that objects that are farther away are irradiated with visible light. On the other hand, for example, the irradiation control portion <NUM> reduces the threshold Th3 as the speed of the vehicle <NUM> decreases so that objects that are farther away are not irradiated with visible light. This is because, when the speed of the vehicle <NUM> is high, farther objects must be monitored for collision prevention and the like, but when the speed of the vehicle <NUM> is low, there is not such a need for monitoring farther objects.

Specific examples of a control method of an irradiation pattern will now be described with reference to <FIG>.

<FIG> shows an example of a photographed infrared image of the front of the vehicle <NUM>.

A recognized area R1 is an area that includes a person riding a bicycle. For example, when a recognition score of the recognized area R1 is equal to or higher than a threshold Th1, light intensity of visible light with respect to the recognized area R1 is reduced. On the other hand, when the recognition score of the recognized area R1 is lower than the threshold Th1, the light intensity of visible light with respect to the recognized area R1 is increased. In addition, the light intensity of visible light with respect to the recognized area R1 is changed in accordance with, for example, a distance to the person inside the area.

A recognized area R2 is an area that includes a traffic sign with a reflective plate. For example, light intensity of visible light with respect to the recognized area R2 is reduced.

In addition, the light intensity of visible light with respect to the recognized area R2 is changed in accordance with, for example, a distance to the traffic sign inside the area.

A recognized area R3 is an area that includes a traffic light that emits light. For example, light intensity of visible light with respect to the recognized area R3 is reduced. In addition, the light intensity of visible light with respect to the recognized area R3 is changed in accordance with, for example, a distance to the traffic light inside the area.

A recognized area R4 is an area that includes another vehicle. For example, when a recognition score of the recognized area R4 is equal to or higher than the threshold Th1, light intensity of visible light with respect to the recognized area R4 is reduced. On the other hand, when the recognition score of the recognized area R4 is lower than the threshold Th1, the light intensity of visible light with respect to the recognized area R4 is increased. In addition, the light intensity of visible light with respect to the recognized area R4 is changed in accordance with, for example, a distance to the vehicle inside the area.

A recognized area R5 is an area that includes a distant building and the sky. For example, light intensity of visible light with respect to the recognized area R5 is set to <NUM>.

The recognized area R6 is an extremely dark area which is close to the vehicle <NUM> and of which a recognition score is extremely low. For example, light intensity of visible light with respect to the recognized area R6 is increased in order to improve the recognition score.

<FIG> shows an example of a result of semantic segmentation performed with respect to a visible image with a same angle of view as in <FIG>.

A road surface is present in a recognized area R11. Therefore, control is performed so that the recognized area R11 is constantly irradiated with visible light. Light intensity of visible light with respect to the recognized area R11 is changed in accordance with, for example, a distance to the road surface inside the area.

A recognized area R12 is an area that includes a person riding a bicycle. For example, when a recognition score of the recognized area R12 is equal to or higher than the threshold Th1, light intensity of visible light with respect to the recognized area R12 is reduced. On the other hand, when the recognition score of the recognized area R12 is lower than the threshold Th1, the light intensity of visible light with respect to the recognized area R12 is increased. In addition, the light intensity of visible light with respect to the recognized area R12 is changed in accordance with, for example, a distance to the person inside the area.

A recognized area R13 is an area that includes a traffic sign. For example, light intensity of visible light with respect to the recognized area R13 is reduced. In addition, the light intensity of visible light with respect to the recognized area R13 is changed in accordance with, for example, a distance to the traffic sign inside the area.

A recognized area R14 is an area that includes a traffic light. For example, light intensity of visible light with respect to the recognized area R14 is reduced. In addition, the light intensity of visible light with respect to the recognized area R14 is changed in accordance with, for example, a distance to the traffic light inside the area.

Recognized areas R15 to R17 are areas that include another vehicle. For example, light intensity of visible light is reduced with respect to an area with a recognition score that is equal to or higher than the threshold Th1 among the recognized areas R15 to R17. On the other hand, light intensity of visible light is increased with respect to an area with a recognition score that is lower than the threshold Th1 among the recognized areas R15 to R17. In addition, the light intensity of visible light with respect to the recognized areas R15 to R17 is changed in accordance with, for example, a distance to the vehicle inside the areas.

The recognized area R18 is an extremely dark area which is close to the vehicle <NUM> and of which a recognition score is extremely low. For example, light intensity of visible light with respect to the recognized area R18 is increased in order to improve the recognition score.

<FIG> shows another example of a photographed infrared image of the front of the vehicle <NUM>.

The recognized area R21 is an extremely dark area which is close to the vehicle <NUM> and of which a recognition score is extremely low. For example, light intensity of visible light with respect to the recognized area R21 is increased in order to improve the recognition score.

Recognized areas R22 and R23 are areas that include a person. For example, light intensity of visible light is reduced with respect to an area with a recognition score that is equal to or higher than the threshold Th1 among the recognized areas R22 and R23. On the other hand, light intensity of visible light is increased with respect to an area with a recognition score that is lower than the threshold Th1 among the recognized areas R22 and R23. In addition, the light intensity of visible light with respect to the recognized areas R22 and R23 is changed in accordance with, for example, a distance to the person inside the areas.

Returning to <FIG>, the processing subsequently returns to step S2 and processing of step S2 and thereafter is executed.

As described above, by appropriately controlling an irradiation pattern of visible light, accuracy of image recognition processing with respect to a visible image improves. As a result, for example, recognition accuracy of an object in the visible image improves and, in turn, performance of functions (for example, automatic driving, ADAS, and the like) of the vehicle <NUM> using the recognition result improves.

Hereinafter, modifications of the embodiment of the present technique described above will be described.

While an example in which semantic segmentation is used as image recognition processing has been presented in the description given above, the present technique can also be applied when using other methods of image recognition processing.

In addition, for example, the present technique can also be applied when image recognition processing is performed using only a visible image. In this case, for example, an irradiation pattern is controlled based on a recognition score of each recognized area of the visible image, a type of an object, and a distance to the object according to a similar method to a case where a result of image recognition processing with respect to an infrared image is used.

Furthermore, in the control processing of an irradiation pattern described above, recognized areas may be created in which light intensity of visible light is reduced or set to <NUM>. This is not particularly problematic in a case of automatic driving that does not involve a driver. On the other hand, when a driver is driving, visibility of the driver may deteriorate. Therefore, when a driver is driving, an irradiation pattern may be controlled by the method described above only during a period in which image recognition processing is performed, and the front of the vehicle <NUM> may be uniformly illuminated with visible light during a period in which image recognition processing is not performed.

In addition, while an example in which image recognition processing is performed with respect to an image of the front of the vehicle <NUM> has been presented in the description given above, the present technique can also be applied when performing image recognition processing with respect to an image in a direction other than the front (for example, the side, the rear, above, or the like). In this case, for example, an irradiation pattern of visible light in a photographing direction of each image is controlled.

Furthermore, the present technique can also be applied to a mobile body other than a vehicle. For example, the present technique can also be applied to mobile bodies such as personal mobility, construction machinery, and agricultural and farm machinery (tractors). In addition, the present technique can also be applied to mobile bodies that are remotely operated (manipulated) without carrying a user such as a drone and a robot as well as mobile bodies that are capable of automatic driving.

In addition, the present technique can also be applied when image recognition processing is performed with respect to an image other than an image photographed from a mobile body. For example, the present technique can also be applied when image recognition processing is performed with respect to a monitoring image photographed by a camera installed in a periphery of a road.

Furthermore, all of the information processing portion <NUM> shown in <FIG> need not necessarily be realized by the integrated control unit <NUM> and a part of or all of the information processing portion <NUM> may be realized by one or more other control units.

In addition, a part of or all of the information processing portion <NUM> may be provided in, for example, the variable irradiation lamp <NUM> or the visible light camera <NUM>.

The series of processing described above can be executed by hardware or by software.

The program to be executed by the microcomputer <NUM> shown in <FIG> or the like may be a program which causes processing to be time-sequentially performed along an order described in the present specification or a program which causes processing to be performed in parallel or at necessary timings such as when a call is made.

In addition, in the present specification, a system signifies a set of a plurality of components (apparatuses, modules (parts), and the like), and whether or not all of the components are present inside a same casing does not matter. Therefore, a plurality of apparatuses which are housed in separate casings but which are connected to each other via a network and a single apparatus in which a plurality of modules are housed in a single casing are both considered systems.

Furthermore, embodiments of the present technique are not limited to the embodiment described above and various modifications can be made without departing from the gist of the present technique.

For example, the present technique may adopt a configuration of cloud computing in which a single function is shared among and cooperatively processed by a plurality of apparatuses via a network.

In addition, each step explained in the flow charts described above can be executed in a shared manner by a plurality of apparatuses in addition to being executed by a single apparatus.

Claim 1:
An information processing apparatus, comprising:
a recognizing portion (<NUM>) configured to perform image recognition processing with respect to an image photographed by a photographing portion and calculate, for each recognized area, the recognized area corresponding to an object recognized, a reliability of the recognition result; and
an irradiation control portion (<NUM>) configured to control an irradiation pattern of visible light in a photographing direction of the photographing portion and control light intensity of visible light with respect to a low-reliability area based on the reliability and at least one of a distance to an object and a speed of a vehicle where the information processing apparatus is installed,
wherein the low-reliability area is the recognized area of which the reliability is lower than a prescribed threshold,
wherein the irradiation control portion (<NUM>) is configured to set the light intensity to zero when the distance to the object is equal or greater than a distance threshold, wherein the irradiation control portion (<NUM>) is configured to increase the distance threshold as the speed of the vehicle increases.