Patent Description:
Traditional solutions for image collection related to a vehicle include two main types. The first type is that, for example, a photographer in the vehicle photographs an image outside a vehicle by using a mobile device (e.g., a mobile phone). The second type is that, for example, an image is collected by using cameras (e.g., a rear view camera) equipped in the vehicle and is displayed on an in-vehicle display to assist in parking. For the first solution for image collection, it is hard to collect the image expected by the photographer, such as a surrounding image of a roof, a rear, or a side of the vehicle, due to being sheltered by a vehicle body or a limitation of a photographing angle in the vehicle. While for the second solution for image collection, current cameras equipped in the vehicle cannot be informed with a photographing intention of the photographer, and are difficult to cooperatively collect the surrounding image that meets expectations of the photographer, because the cameras equipped in the vehicle are mainly used for driving assistance and are respectively equipped in multiple different locations of the vehicle, including, for example, an interior view camera, a rear view camera, a front view camera, a side view camera, and so on.

Therefore, in the traditional solutions for image collection, it is impossible to collect images that are not limited by the photographing angle in the vehicle, according to the photographing intention of the photographer in the vehicle, because of many reasons such as the shelter by the vehicle body and installation location limitations of the cameras.

<CIT> discloses a remote control method and apparatus for controlling the state of a movable object and/or a load carried thereon. The remote control method comprising: receiving, via an apparatus, a state signal that corresponds to a user's position; remote-controlling the state of the load being carried on a movable object based on the state signal; wherein the state of the load is the result of combining the movement of the load relative to the movable object and the movement of the object relative to its environment. For example, the control of the state can be achieved through the state of the apparatus itself, a user's state captured by an apparatus, a graphical interface on a screen of an apparatus, or a voice command.

<NPL> provides a full auto-calibration of a smartphone on board a vehicle using IMU and GPS embedded sensors. In particular, the paper proposes an automatic method to calibrate a smartphone on board a vehicle using its embedded IMU and GPS, based on longitudinal vehicle acceleration. The method estimates the yaw angle of a smartphone relative to a vehicle in every case, even on non-zero slope roads. Furthermore, in order to decrease the impact of IMU noise, an algorithm based on Kalman Filter and fitting a mixture of Gaussians is introduced. The results show that the system achieves high accuracy, the typical error is <NUM>%, and is immune to electromagnetic interference.

The foregoing and other features, advantages, and aspects of implementations of the present disclosure will become more apparent in conjunction with the accompanying drawings and with reference to the following detailed description. In the accompanying drawings of the present disclosure, the same or similar accompanying drawings numerals generally represent the same or similar elements.

Preferred implementations of the present disclosure will be described in more details with reference to the drawings. Although the drawings illustrate the preferred implementations of the present disclosure, it should be appreciated that the present disclosure can be implemented in various manners and should not be limited to the implementations explained herein. On the contrary, the implementations are provided to make the present disclosure more thorough and complete and to fully convey the scope of the present disclosure to those skilled in the art.

As used herein, the term "include" and its variants are to be read as open-ended terms that mean "include, but is not limited to. " The term "or" is to be read as "and/or" unless the context clearly indicates otherwise. The term "based on" is to be read as "based at least in part on. " The terms "one example implementation" and "one implementation" are to be read as "at least one example implementation. " The term "a further implementation" is to be read as "at least a further implementation. " The terms "first", "second" and so on can refer to same or different objects. The following text also can include other explicit and implicit definitions.

As mentioned above, in the above-mentioned related solutions for image collection, it is impossible to collect images that are not limited by the photographing angle in the vehicle, according to the photographing intention of the photographer in the vehicle, because of many reasons such as the shelter by the vehicle body and installation location limitations.

To solve at least partly the foregoing problem and one or more of other potential problems, a method for image collection is provided according to example implementations of the disclosure. The method includes the following. An initial photographing direction of a mobile device associated with a vehicle is obtained, in response to determining that a predetermined condition is satisfied. An in-vehicle photography apparatus of the vehicle is adjusted to an initial orientation, to match a photographing direction at the initial orientation of the in-vehicle photography apparatus with the initial photographing direction of the mobile device. A drive signal configured to adjust the photographing direction of the in-vehicle photography apparatus is generated, based on a detecting signal of a sensor of the mobile device, such that the photographing direction of the in-vehicle photography apparatus synchronously varies with a pose of the mobile device. A surrounding image collected via the in-vehicle photography apparatus is transmitted to the mobile device, to display the surrounding image at the mobile device.

In the above-mentioned method, the in-vehicle photography apparatus is adjusted to the initial orientation matched with the initial photographing direction of the mobile device by determining that the predetermined condition is satisfied. The drive signal is generated, based on the detecting signal of the sensor of the mobile device, such that the photographing direction of the in-vehicle photography apparatus synchronously varies with the pose of the mobile device. Further, the surrounding image collected via the in-vehicle photography apparatus is transmitted to the mobile device. Therefore, in the disclosure, it is possible to adjust a photographing angle of the in-vehicle photography apparatus synchronously by adjusting an orientation a mobile phone, and the mobile device may display the surrounding image collected via the synchronously adjusted in-vehicle photography apparatus. Therefore, it is also possible to collect the surrounding image outside the vehicle according to a photographing intention of a photographer in the vehicle, and the image collected is not limited by a photographing field of view in the vehicle.

<FIG> is a schematic diagram illustrating a system <NUM> of a method for image collection according to implementations of the disclosure. As illustrated in <FIG>, the system <NUM> includes a vehicle <NUM>, a mobile device <NUM>, and a server <NUM>. In some examples, the vehicle <NUM>, the mobile device <NUM> of a user <NUM> (e.g., a passenger), and the server <NUM>, for example, may perform data interaction via a base station <NUM> or a network <NUM>. The vehicle <NUM> and the mobile device <NUM> may also perform data interaction and sharing via wireless communication methods, such as wireless fidelity (Wi-Fi), Bluetooth, cellular, and near field communication (NFC).

The vehicle <NUM> includes for example at least an in-vehicle calculating device <NUM> (such as a video audio entertainment system (VAES)), an in-vehicle data sensing device, an in-vehicle telematics BOX (T-BOX), and the like. The in-vehicle data sensing device is configured to perceive vehicle data of its own and external environment data where the vehicle is located in real time. The in-vehicle data sensing device includes at least multiple in-vehicle photography apparatuses <NUM>, such as a front view camera, a rear view camera, a roof-mounted photography apparatus, and the like. The front view camera is configured to collect a surrounding image of the front of the vehicle and the rear view camera is configured to collect a surrounding image of the rear of the vehicle. The roof-mounted photography apparatus is configured to adjust a photographing direction based on a received drive signal, such that the photographing direction of the roof-mounted photography apparatus covers a panoramic view of the surrounding image of the external environment of the vehicle. The vehicle <NUM> and the mobile device <NUM> may perform data interaction and sharing via wireless communication methods such as Wi-Fi, Bluetooth, cellular, and NFC. For example, the mobile device <NUM> can establish an association with the vehicle <NUM> when a predetermined action (e.g., shaking) of the mobile device <NUM> is detected. By establishing the association between the mobile device <NUM> and the vehicle <NUM> through the predetermined action (e.g., shaking), it is possible to establish a connection between the vehicle and an associated mobile device of a specific user (e.g., a driver) in a secure manner, so as to share data and computing resources.

The in-vehicle T-BOX is configured to perform data interaction with the in-vehicle calculating device <NUM> (such as the VAES), the mobile device <NUM>, and the server <NUM>. In some examples, the in-vehicle T-BOX includes, for example, a subscriber identity module (SIM) card, a global position system (GPS) antenna, a 4th generation (<NUM>) antenna, or a 5th generation (<NUM>) antenna. When the user sends a control command (such as remotely starting the vehicle, opening an air conditioning, adjusting a seat to a proper position) via an application (APP) of the mobile device <NUM> (such as a mobile phone), a telematics service provider (TSP) will send in background a monitoring request instruction to the in-vehicle T-BOX. After the vehicle obtains the control command, it sends a control packet and implements control of the vehicle via a controller area network (CAN) bus, and finally feeds back an operation result to the APP of the mobile device <NUM> of the user. The in-vehicle T-BOX and the VAES communicate via the CAN bus to implement data interaction, such as transmitting state information of the vehicle, button state information of the vehicle, a control instruction. The in-vehicle T-BOX may collect bus data related to a bus of the vehicle <NUM> such as a diagnose CAN (DCAN), a kommunikation CAN (KCAN), and a power train CAN (PTCAN).

The roof-mounted photography apparatus, in some examples, includes, for example, a camera, a first rotation apparatus, a second rotation apparatus, and an elevating apparatus. The first rotation apparatus is configured to drive the camera to rotate <NUM> degrees around a first axis. The second rotation apparatus is configured to drive the camera to rotate (a rotation angle range, for example, between <NUM> and <NUM> degrees) around a second axis, where the second axis is perpendicular to the first axis. The elevating apparatus is configured to drive the camera to move in a vertical direction. In some examples, the elevating apparatus is configured to extend the roof-mounted photography apparatus out from inside of the vehicle, or retract the roof-mounted photography apparatus from outside of the vehicle. The roof-mounted photography apparatus may perform data interaction and sharing with the mobile device <NUM> via the vehicle <NUM> (e.g., the in-vehicle calculating device <NUM> and/or the in-vehicle T-BOX). The roof-mounted photography apparatus may also perform data interaction and sharing directly with the mobile device <NUM> via wireless communication methods such as Wi-Fi, Bluetooth, cellular, and NFC. In some examples, the roof-mounted photography apparatus further includes a NFC module configured to perform a short-range communication with electronic devices, such as the mobile device <NUM>. NFC has a communication distance within tens of centimeters, an operating frequency of <NUM>, and a transmission speed which is, for example, but not limited to <NUM> Kbit/s, <NUM> Kbit/s, or <NUM> Kbit/s. The roof-mounted photography apparatus may conveniently and safely exchange data with the mobile device <NUM> based on the NFC module, when the mobile device <NUM> touches or approaches the roof-mounted photography apparatus (e.g., the mobile device <NUM> touches a roof of the vehicle near the roof-mounted photography apparatus), where the exchanged data is validation information, for example, a Wi-Fi password. The mobile device <NUM> may obtain the surrounding image collected via the roof-mounted photography apparatus, based on the validation information obtained, when the predetermined condition is satisfied. In some examples, the vehicle <NUM> may also equip one or more NFC modules at other predetermined locations in the vehicle (e.g., at a vehicle door), to facilitate passengers at different locations in the vehicle to conveniently obtain the validation information by approaching or touching the predetermined locations with the mobile device <NUM>. Since a NFC technology has higher security compared to Bluetooth, ZigBee, infrared, Wi-Fi and other technologies, the NFC technology has performance advantages in the short-range communication, and the NFC technology has lower costs, simple setup procedures, and short communication establishment time of only about <NUM>. Therefore, in the disclosure, it is possible to take into account both a low energy consumption of information interaction and information security at the same time, by obtaining the validation information based on the NFC technology.

The mobile device <NUM> is, for example, but not limited to, a mobile phone. The mobile device <NUM> may perform data interaction directly with the in-vehicle T-BOX, or with the server <NUM> via the base station <NUM> or the network <NUM>. In some examples, the mobile device <NUM> may be a tablet computer, a mobile phone, a wearable device, and the like.

The server <NUM> is configured to, for example, provide services of internet of vehicles. The server <NUM>, for example, performs data interaction with the vehicle <NUM> and the mobile device <NUM> via the network <NUM> or the base station <NUM>. In some examples, the server <NUM> may have one or more processing units, including dedicated processing units such as graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and application specific integrated circuits (ASICs), and general-purpose processing units such as central processing units (CPUs). In addition, one or more virtual machines may also be running on each computing device.

In the following, a method for image collection of implementations of the disclosure will be described in conjunction with <FIG> is a schematic flow chart illustrating a method <NUM> for image collection according to implementations of the disclosure. It should be understood that, for example, the method <NUM> may be performed by the electronic device <NUM> as shown in <FIG> or may be performed by the mobile device <NUM> or the vehicle <NUM> (for example, but not limited to the in-vehicle calculating device <NUM> such as the VAES) as shown in <FIG>. It should be noted that the method <NUM> may further include additional actions not shown and/or may omit actions shown, and the disclosure is not limited thereto.

At block <NUM>, the in-vehicle calculating device <NUM> determines whether a predetermined condition is satisfied. In some implementations, the in-vehicle calculating device <NUM> determines that the predetermined condition is satisfied as follows. The in-vehicle calculating device <NUM> determines that verification of validation information from the mobile device <NUM> already passes, where the validation information is obtained by the mobile device <NUM> via touching a predetermined location of the vehicle <NUM>, and at least one of the following conditions is satisfied. A photographing direction of the mobile device <NUM> coincides with a travel direction of the vehicle <NUM>, or a predetermined action at the mobile device <NUM> is detected. The mobile device <NUM>, for example, obtains the validation information (e.g., a Wi-Fi password) via touching the predetermined location (e.g., a door or a roof of the vehicle) of the vehicle <NUM>, and the in-vehicle calculating device <NUM> determines that the verification of the validation information transmitted by the mobile device <NUM> passes. The in-vehicle calculating device <NUM> determines that the predetermined condition is satisfied in a case that the in-vehicle calculating device <NUM> further detects that the photographing direction of the mobile device <NUM> coincides with the travel direction of the vehicle <NUM> (e.g., a Z axis of the mobile device <NUM> is parallel to the travel direction of the vehicle <NUM>, or a direction perpendicular to a display of the mobile device <NUM> is parallel to a longitudinal direction of the vehicle <NUM>), or determines that the predetermined action at the mobile device <NUM> is detected.

At block <NUM>, the in-vehicle calculating device <NUM> obtains an initial photographing direction of the mobile device <NUM> associated with the vehicle <NUM>, in response to determining that the predetermined condition is satisfied.

For example, the in-vehicle calculating device <NUM> obtains the initial photographing direction of the mobile device <NUM> at a time point t0, when the in-vehicle calculating device determines that the predetermined condition is satisfied. For example, the in-vehicle calculating device obtains detection information of a gyro-sensor and an acceleration sensor of the mobile device <NUM> at the time point t0, e.g., at the time point t0, angular velocities of the mobile device <NUM> at three axes (an X axis, a Y axis, and the Z axis of the mobile device <NUM>) are respectively αx, αy, and αz. The X axis of the mobile device <NUM> is a width direction of the mobile device <NUM>. When the top of the mobile device <NUM> points upwards, an X-axis positive direction is along the display of the mobile device <NUM> to the right, and an X-axis negative direction is along the display of the mobile device <NUM> to the left. The Y axis is the longitudinal direction of the mobile device <NUM>. A Y-axis positive direction points upwards from the top of the display of the mobile device <NUM>, and a Y-axis negative direction points downwards from that of the mobile device <NUM>. The Z axis is the direction perpendicular to the display of the mobile device <NUM>. A Z-axis positive direction is perpendicular to the display and points outwards, and a Z-axis negative direction is perpendicular to the display and points inwards.

In some implementations, the in-vehicle calculating device <NUM> may also obtain an image feature(s) in the image collected by the mobile device <NUM> at the initial photographing direction to identify the initial photographing direction of the mobile device <NUM>. For example, the image collected by the mobile device is obtained, when the in-vehicle calculating device <NUM> determines that a photography apparatus of the mobile device <NUM> is already turned on, and the image feature(s) in the image collected by the mobile device <NUM> is extracted to identify the initial photographing direction of the mobile device <NUM>.

At block <NUM>, the in-vehicle calculating device <NUM> adjusts the in-vehicle photography apparatus <NUM> of the vehicle <NUM> to the initial orientation, to match the photographing direction at the initial orientation of the in-vehicle photography apparatus <NUM> with the initial photographing direction of the mobile device <NUM>. In some implementations, the matching means that, for example, photographing direction values of the in-vehicle photography apparatus <NUM> and the mobile device <NUM> are approximately the same, or a photographing direction deviation of the in-vehicle photography apparatus <NUM> and the mobile device <NUM> is within a predetermined threshold range.

In some implementations, in order to determine whether the in-vehicle photography apparatus <NUM> (e.g., a roof-mounted photography apparatus) is already adjusted to the initial orientation, the in-vehicle calculating device <NUM> may obtain the image feature(s) in the image collected by the mobile device <NUM> at the initial photographing direction, and the in-vehicle calculating device may obtain a surrounding image collected via the roof-mounted photography apparatus. It is determined that the roof-mounted photography apparatus is already adjusted to the initial orientation, when the in-vehicle calculating device determines that the surrounding image collected via the roof-mounted photography apparatus is matched with the image feature(s) in the image collected by the mobile device <NUM> at the initial photographing direction.

In some implementations, indication information is transmitted to the mobile device <NUM>, when the in-vehicle calculating device <NUM> determines that the in-vehicle photography apparatus <NUM> (e.g., the roof-mounted photography apparatus) is already adjusted to the initial orientation. The in-vehicle photography apparatus <NUM> is controlled to photograph or record the surrounding image, when the in-vehicle calculating device <NUM> determines that a predetermined operation at the mobile device <NUM> or a predetermined voice input is detected. In the disclosure, by using the above-mentioned method, it is possible to conveniently control a photographing action of the in-vehicle photography apparatus <NUM>, by recognizing a user voice in the vehicle or a user operation on the mobile phone.

The in-vehicle photography apparatus <NUM> (e.g., the roof-mounted photography apparatus), in some implementations, is a panoramic photography apparatus equipped at the roof of the vehicle <NUM>. The panoramic photography apparatus at least includes a camera, a first rotation apparatus configured to drive the camera to rotate around a first axis, a second rotation apparatus configured to drive the camera to rotate around a second axis (where the second axis is perpendicular to the first axis), and an elevating apparatus configured to drive the camera to move in a vertical direction. The first rotation apparatus is configured to, for example, rotate driven by a first drive signal, and the second rotation apparatus is configured to, for example, rotate driven by a second drive signal. In some implementations, the in-vehicle photography apparatus <NUM> further includes an NFC module configured to cause the mobile device <NUM> to obtain validation information for verification of the mobile device <NUM> via touching or approaching the in-vehicle photography apparatus <NUM>. In the following, a structure and a drive mechanism of the in-vehicle photography apparatus <NUM> (e.g., the roof-mounted photography apparatus) will be described in detail in conjunction with <FIG>, which will not be repeated herein.

The in-vehicle calculating device <NUM> can adjust the in-vehicle photography apparatus <NUM> of the vehicle <NUM> to the initial orientation in various ways. For example, the in-vehicle calculating device <NUM> may determine the initial orientation of the in-vehicle photography apparatus <NUM> based on location information of an internal component of the vehicle <NUM> and the image feature(s), if the in-vehicle calculating device <NUM> determines that the image feature(s) in the image collected by the mobile device <NUM> at the initial photographing direction is associated with the internal component. In the following, a method for adjusting the in-vehicle photography apparatus <NUM> to the initial orientation will be described in detail in conjunction with <FIG>, which will not be repeated herein.

At block <NUM>, the in-vehicle calculating device <NUM> generates a drive signal configured to adjust the photographing direction of the in-vehicle photography apparatus <NUM>, based on a detecting signal of a sensor of the mobile device <NUM>, such that the photographing direction of the in-vehicle photography apparatus <NUM> synchronously varies with a pose of the mobile device <NUM>. In some examples, the detecting signal of the sensor of the mobile device <NUM> is at least one of, for example, an acceleration, an angular velocity, or a magnetic field data obtained by an acceleration sensor, the gyro-sensor, and an electronic compass of the mobile device <NUM>.

There are various methods for generating the drive signal configured to adjust the photographing direction of the in-vehicle photography apparatus <NUM>. In some implementations, the in-vehicle calculating device <NUM> may obtain pose angle change information of a current orientation of the mobile device <NUM> relative to an orientation corresponding to the initial photographing direction, and may generate the drive signal configured to drive at least one of the first rotation apparatus or the second rotation apparatus to rotate, based on the pose angle change information. The drive signal includes at least one of the first drive signal or the second drive signal. The first drive signal is configured to drive the first rotation apparatus of the in-vehicle photography apparatus <NUM> (e.g., the roof-mounted photography apparatus), and the second drive signal is configured to drive the second rotation apparatus of the in-vehicle photography apparatus <NUM>. In the following, a method for generating the drive signal configured to adjust the photographing direction of the in-vehicle photography apparatus <NUM>, based on a detecting signal of a pose sensor of the mobile device <NUM>, will be described in detail in conjunction with <FIG> and <FIG>, which will not be repeated herein.

The method for generating the drive signal configured to adjust the photographing direction of the in-vehicle photography apparatus <NUM>, in some implementations, is described as follows. The in-vehicle calculating device <NUM> can determine whether the detecting signal of the pose sensor of the mobile device <NUM> and a detecting signal of the acceleration sensor of the mobile device <NUM> are both in a predetermined range. The in-vehicle calculating device <NUM> can generate the drive signal configured to adjust the photographing direction of the in-vehicle photography apparatus <NUM>, based on the detecting signal of the pose sensor of the mobile device, when the in-vehicle calculating device <NUM> determines that the detecting signal of the pose sensor of the mobile device <NUM> and the detecting signal of the acceleration sensor of the mobile device <NUM> are both in the predetermined range. For example, when the in-vehicle calculating device <NUM> determines that at least one of the detecting signal of the pose sensor of the mobile device <NUM> or the detecting signal of the acceleration sensor of the mobile device <NUM> exceeds the predetermined range, it is determined that the mobile device <NUM> may occur unexpected situations such as shaking, vibrating, dropping, etc., and thus the in-vehicle calculating device <NUM> will not generate the drive signal based on the detection signal of the sensor of the mobile device <NUM> in the above-mentioned unexpected situations. By using the above-mentioned method, it is possible to prevent the in-vehicle photography apparatus <NUM> from being incorrectly adjusted due to unexpected situations of the mobile device <NUM> such as shaking, vibrating, and dropping.

At block <NUM>, the in-vehicle calculating device <NUM> transmits the surrounding image collected via the in-vehicle photography apparatus <NUM> to the mobile device <NUM>, to display the surrounding image at the mobile device <NUM>. The image collected via the in-vehicle photography apparatus <NUM> may be a photo or video data.

A method for transmitting the surrounding image, in some implementations, is described as follows. The in-vehicle photography apparatus <NUM> can collect the surrounding image outside the vehicle in real time and form the video data, process and package the video data, and send the packaged video data to the in-vehicle calculating device <NUM>. The in-vehicle calculating device <NUM> can transmit the received packaged video data to the mobile device <NUM> via Wi-Fi or a universal serial bus (USB) interface. The mobile device <NUM> can unpack and process the received packaged data, for reconstructing a complete video data stream for displaying on the display. In some implementations, the in-vehicle photography apparatus <NUM> can also directly transmit the packaged video data to the mobile device <NUM> via Wi-Fi.

In some examples, the mobile device <NUM> can also control the in-vehicle photography apparatus <NUM> (e.g., the roof-mounted photography apparatus) to photograph or capture images, move (e.g., elevate, rotate along the vertical direction), adjust the photographing field of view, and other operations, directly through predetermined operations on the display (e.g., click-to-select, sliding, zooming, and other actions) or via the in-vehicle calculating device <NUM>.

In some examples, the in-vehicle calculating device <NUM> may also transform the surrounding image collected via the in-vehicle photography apparatus <NUM> to fit the display of the mobile device.

In the above-mentioned method, the in-vehicle photography apparatus <NUM> is adjusted to the initial orientation matched with the initial photographing direction of the mobile device by determining that the predetermined condition is satisfied. The drive signal is generated, based on the detecting signal of the sensor of the mobile device, such that the photographing direction of the in-vehicle photography apparatus <NUM> synchronously varies with the pose of the mobile device. Further, the surrounding image collected via the in-vehicle photography apparatus <NUM> is transmitted to the mobile device. Therefore, in the disclosure, it is possible to adjust a photographing angle of the in-vehicle photography apparatus <NUM> synchronously by adjusting an orientation of a mobile phone, and the mobile device may display the surrounding image collected via the synchronously adjusted in-vehicle photography apparatus <NUM>. Therefore, it is also possible to collect images according to a photographing intention of a photographer in the vehicle, without limitations of the photographing angle in the vehicle.

<FIG> is a schematic diagram illustrating an in-vehicle photography apparatus <NUM> according to implementations of the disclosure. It should be noted that the in-vehicle photography apparatus <NUM> may further include additional structures not shown and/or may omit structures shown, and the disclosure is not limited thereto.

As illustrated in <FIG>, the in-vehicle photography apparatus <NUM>, for example, includes a camera <NUM>, a first rotation apparatus <NUM>, a second rotation apparatus <NUM>, and an elevating apparatus <NUM>.

The first rotation apparatus <NUM> may rotate from <NUM> to <NUM> degrees around a first axis (that is, a vertical axis perpendicular to a horizontal plane, such as the Z axis) in a first plane (e.g., the horizontal plane). In some examples, a rotation range of the first rotation apparatus <NUM> may also be less than <NUM> degrees. The first rotation apparatus <NUM> is, for example, connected with a rotary shaft of a first drive source (not shown). As shown in <FIG>, the first rotation apparatus <NUM> may also be driven to rotate by a rotary shaft <NUM> of the first drive source (e.g., a first motor <NUM>) and a first drive mechanism (e.g., a gear or a transmission belt <NUM>). In some examples, a rotation angle of the first rotation apparatus <NUM> is controlled by the first drive signal.

The second rotation apparatus <NUM> may rotate from <NUM> to <NUM> degrees around a second axis (e.g., a horizontal axis parallel to the first plane and perpendicular to the first axis). In some examples, the second rotation apparatus <NUM> may also rotate less than <NUM> degrees, such as rotate in a clockwise direction as shown by an arrow <NUM> in <FIG> or in a counterclockwise direction. The second rotation apparatus <NUM> is, for example, a second drive source (e.g., a second motor, which includes a rotor and a stator connected with the rotary shaft), and the rotary shaft of the second rotation apparatus <NUM> may be connected with the camera <NUM> directly or via a second drive mechanism (e.g., the gear). The photographing direction of the camera <NUM> rotates along with rotation of the rotary shaft of the second rotation apparatus <NUM>. In some examples, a rotation angle of the second rotation apparatus <NUM> is controlled by the second drive signal. The second rotation apparatus <NUM> has a fixed portion (e.g., a housing of the second motor) fixedly connected with a support device <NUM>.

The housing of the second rotation apparatus <NUM> is relatively fixedly connected with the first rotation apparatus <NUM> via the support device <NUM>. Since the housing of the second rotation apparatus <NUM> is relatively fixedly connected with the first rotation apparatus <NUM>, and the rotary shaft of the first rotation apparatus <NUM> is connected with the camera <NUM>, when the first drive source (e.g., the first motor <NUM>) drives the first rotation apparatus <NUM> to rotate a predetermined angle around the Z axis, the first rotation apparatus <NUM> can also drive the camera <NUM> to rotate the predetermined angle around the Z axis.

By using the above-mentioned method, the first rotation apparatus <NUM> can drive the camera <NUM> to rotate around the Z axis (the vertical axis, i.e., the first axis) perpendicular to the horizontal plane, and the second rotation apparatus <NUM> can drive the camera <NUM> to rotate around the second axis perpendicular to the first axis.

In some examples, the elevating apparatus <NUM> of the in-vehicle photography apparatus <NUM> can drive the in-vehicle photography apparatus <NUM> to go up or down along the vertical direction, to facilitate the in-vehicle photography apparatus <NUM> to extend out from inside of the vehicle or retract from outside of the vehicle.

As illustrated in <FIG>, the in-vehicle photography apparatus <NUM>, for example, includes a camera <NUM>, a first rotation apparatus <NUM>, a second rotation apparatus <NUM>, and an elevating apparatus (not shown).

The first rotation apparatus <NUM> may rotate from <NUM> to <NUM> degrees around the first axis (an axis perpendicular to a first plane) in the first plane (e.g., a plane where the first rotation apparatus <NUM> is located). The first rotation apparatus <NUM> is, for example, connected with a rotary shaft of a first drive source (not shown). As shown in <FIG>, the first rotation apparatus <NUM> may also be driven to rotate by a rotary shaft <NUM> of the first drive source (e.g., a first motor <NUM>) and a first drive mechanism (e.g., a gear or a transmission belt <NUM>). The camera <NUM> is fixedly connected with the first rotation apparatus <NUM> via a support device <NUM>. The photographing direction of the camera <NUM> rotates along with rotation of the first rotation apparatus <NUM>. The rotation angle of the first rotation apparatus <NUM> is, for example, controlled by the first drive signal. For example, the photographing direction of the camera <NUM> is adjusted a yaw due to the rotation of the first rotation apparatus <NUM> driven by the first drive signal.

The second rotation apparatus <NUM> may rotate from <NUM> to <NUM> degrees around the second axis (e.g., an axis parallel to the first plane). The second axis is perpendicular to the first axis. In some examples, the second rotation apparatus <NUM> may also rotate less than <NUM> degrees. The second rotation apparatus <NUM> is, for example, connected with a rotation shaft <NUM> of a second drive source (not shown). In some examples, the rotation angle of the second rotation apparatus <NUM> is controlled by the second drive signal. For example, the photographing direction of the camera <NUM> is adjusted a pitch due to the rotation of the second rotation apparatus <NUM> driven by the second drive signal.

In the following, a method for generating the drive signal configured to adjust the photographing direction of the in-vehicle photography apparatus <NUM>, based on the detecting signal of a pose sensor of a mobile device <NUM>, will be described in detail in conjunction with <FIG>, <FIG> and <FIG> is a schematic diagram illustrating a method for generating the drive signal according to implementations of the disclosure.

As illustrated in <FIG>, the initial photographing direction of the mobile device <NUM> at the time point t0 is obtained, when the in-vehicle calculating device <NUM> of the vehicle <NUM> determines that the predetermined condition is satisfied. For example, at the time point t0, angular velocities of the mobile device <NUM> at three axes obtained by the in-vehicle calculating device <NUM> are respectively wx, wy, and wz.

For example, at a time point t1, the user <NUM> rotates the mobile device <NUM> to a direction different from the direction at the time point t0. For example, at the time point t1, the angular velocities of the mobile device <NUM> at three axes obtained by the in-vehicle calculating device <NUM> are respectively wx', wy', and wz'. In order to make the photographing direction of the camera <NUM> synchronously vary with the pose of the mobile device <NUM>, the in-vehicle calculating device <NUM> can, for example, integrate the angular velocity collected by the gyro-sensor of the mobile device <NUM>, for calculating an offset angle of the gyro-sensor in a sampling duration between the time point t1 and the time point t0, based on a sampling rate and a sampling period of the gyro-sensor of the mobile device <NUM>, and an angular velocity collected by the gyro-sensor at an i-th sampling. For example, the in-vehicle calculating device <NUM> calculates that, between the time point t1 and the time point t0, the offset angle of the mobile device <NUM> rotating around the X axis direction is α and the offset angle of the mobile device <NUM> rotating around the Z axis direction is β.

In some examples, the in-vehicle calculating device <NUM> calculates the first drive signal configured to control the rotation angle of the first rotation apparatus <NUM>, based on the offset angle α of the mobile device <NUM> rotating around the X axis direction between the time point t1 and the time point t0. The in-vehicle calculating device <NUM> further calculates the second drive signal configured to control the rotation angle of the second rotation apparatus <NUM>, based on the offset angle β of the mobile device <NUM> rotating around the Z axis direction.

The in-vehicle calculating device <NUM>, for example, drives the first rotation apparatus <NUM> and the second rotation apparatus <NUM> to rotate, based on the first drive signal and the second drive signal, such that the yaw adjusted by the photographing direction of the camera <NUM> is the offset angle α, and the pitch adjusted by the photographing direction of the camera <NUM> is the offset angle β.

In the following, a method for adjusting the in-vehicle photography apparatus to the initial orientation of implementations of the disclosure will be described in conjunction with <FIG> is a schematic flow chart illustrating a method <NUM> for adjusting the in-vehicle photography apparatus <NUM>, <NUM>, or <NUM> to the initial orientation according to implementations of the disclosure. It should be understood that, for example, the method <NUM> may be performed by the electronic device <NUM> as shown in <FIG> or may be performed by the mobile device <NUM> or the vehicle <NUM> (for example, the in-vehicle calculating device <NUM>) as shown in <FIG>. It should be noted that the method <NUM> may further include additional actions not shown and/or may omit actions shown, and the disclosure is not limited thereto.

At block <NUM>, the in-vehicle calculating device <NUM> determines whether the image feature extracted from the image collected by the mobile device <NUM> is associated with the internal component of the vehicle.

At block <NUM>, the in-vehicle calculating device <NUM> recognizes the internal component based on a recognition model, when the in-vehicle calculating device <NUM> determines that the image feature is associated with the internal component of the vehicle <NUM>. The recognition model is trained in advance by using multiple sample image data containing the internal component of the vehicle <NUM>. In some implementations, the in-vehicle calculating device <NUM> trains, based on multiple training samples, the recognition model, where the recognition model is configured to determine categories of objects to-be-recognized (i.e., the internal component of the vehicle) in images to-be-recognized. In some examples, the recognition model is a neural network model, which can be implemented through algorithm models with different network structures. The training samples are, for example, the multiple manually annotated sample image data containing the internal component of the vehicle <NUM>.

At block <NUM>, the in-vehicle calculating device <NUM> determines the initial orientation of the in-vehicle photographing apparatus, based on the location information of the internal component and the image feature. The internal components of the vehicle <NUM> each have a relatively fixed location in the vehicle. Therefore, the in-vehicle calculating device <NUM>, for example, pre-stores the location information of each of the internal components of the vehicle relative to the in-vehicle photography apparatus. The in-vehicle calculating device <NUM> recognizes the internal component contained in the image collected by the mobile device <NUM> via the recognition model. The in-vehicle calculating device <NUM> can determine the drive signal, which is configured to adjust the photographing direction of the in-vehicle photography apparatus to the initial orientation, based on the location information of the internal component and the image feature(s) (e.g., the location of the internal component in the image).

At block <NUM>, the in-vehicle calculating device <NUM> generates the drive signal configured to adjust the photographing direction of the in-vehicle photography apparatus of the vehicle <NUM> to the initial orientation, based on location information of the initial orientation.

In the above-mentioned method, it is possible to quickly and accurately adjust the photographing direction of the in-vehicle photography apparatus to the initial orientation matched with the initial photographing direction of the mobile device, by recognizing the internal component with known relative location of the vehicle based on the image feature(s) extracted in the image collected by the mobile device <NUM>.

Implementations further provide a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores a machine-executable instruction thereon. When executed, the machine-executable instruction causes a machine to perform the method of any of the above implementations of the disclosure.

<FIG> is a schematic block diagram illustrating an electronic device <NUM> applicable to implement implementations of the disclosure. The electronic device <NUM> may be a device configured to perform the methods <NUM> and <NUM> as illustrated in <FIG> and <FIG>. As illustrated in <FIG>, the device <NUM> includes a center processing unit (CPU) <NUM>, which can perform various suitable actions and processing according to computer program instructions which are stored in a read-only memory (ROM) <NUM> or loaded from a storage unit <NUM> to a random access memory (RAM) <NUM>. In the RAM <NUM>, various programs and data required for the operation of the device <NUM> can also be stored. The CPU <NUM>, the ROM <NUM>, and the RAM <NUM> are connected with each other through a bus <NUM>. An input/output (I/O) interface <NUM> is also connected with the bus <NUM>.

Multiple components in the device <NUM> are connected to the I/O interface <NUM>, these components including an input unit <NUM>, an output unit <NUM>, and a storage unit <NUM>. The processing unit <NUM> performs various methods and processing described above, such as the methods <NUM> and <NUM>. In some implementations, the methods <NUM> and <NUM> may be implemented as computer software programs, which are stored in a machine-readable medium, such as the storage unit <NUM>. In some implementations, part or all of the computer programs may be loaded and/or installed on the device <NUM> via the ROM <NUM> and/or the communication unit <NUM>. When the computer programs are loaded into the RAM <NUM> and executed by the CPU <NUM>, one or more operations of the methods <NUM> and <NUM> described above may be executed. Alternatively, in other implementations, the CPU <NUM> may be configured to perform one or more actions of the methods <NUM> and <NUM> via other any suitable methods (such as via the aid of firmware).

It should be further noted that, the present disclosure may be a method, device, system, and/or computer program product. The computer program product may include a computer-readable storage medium storing computer-readable program instructions for executing various aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a wave guide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein may be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.

Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet by using an Internet Service Provider). In some implementations, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, devices (systems), and computer program products according to implementations of the disclosure.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create module for implementing the functions/actions specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein includes an article of manufacture including instructions which implement aspects of the function/action specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/actions specified in the flowchart or block diagram block or blocks.

Claim 1:
A method for image collection, comprising:
obtaining an image collected by a mobile device associated with a vehicle in response to determining that a photography apparatus of the mobile device is already turned on;
in response to determining that a predetermined condition is satisfied, extracting an image feature in the image collected by the mobile device,
wherein the image feature is used to identify an initial photographing direction of the mobile device, and ,
wherein determining that the predetermined condition is satisfied comprises:
determining that verification of validation information from the mobile device already passes, wherein the validation information is obtained by the mobile device via touching or approaching a predetermined location of the vehicle; and
satisfying at least one of:
a photographing direction of the mobile device coinciding with a travel direction of the vehicle; and
determining that a predetermined action at the mobile device is detected;
determining (<NUM>) whether the image feature is associated with an internal component of the vehicle;
recognizing (<NUM>) the internal component based on a recognition model, in response to determining that the image feature is associated with the internal component of the vehicle, wherein the recognition model is trained by using a plurality of sample image data containing the internal component of the vehicle;
determining (<NUM>) an initial orientation of an in-vehicle photography apparatus of the vehicle, based on location information of the internal component and the image feature;
generating (<NUM>, <NUM>) a drive signal configured to adjust the photographing direction of the in-vehicle photography apparatus, based on location information of the initial orientation and a detecting signal of a sensor of the mobile device, such that the photographing direction of the in-vehicle photography apparatus matches with the initial photographing direction of the mobile device at the initial orientation and thereafter synchronously varies with a pose of the mobile device; and
transmitting (<NUM>) a surrounding image collected via the in-vehicle photography apparatus to the mobile device, to display the surrounding image at the mobile device.