Patent Description:
Any discussion of the background art throughout the specification should in no way be considered as an admission that such art is widely known or forms part of common general knowledge in the field.

In monitoring and surveillance systems, it is often necessary to monitor a scene from different perspectives. This is typically achieved by positioning multiple cameras at different positions and orientations throughout the scene. In some applications, such as vehicle and driver monitoring systems, it is advantageous to be able to track and map the positions of objects from the field of view of one camera to another. In these applications, it is necessary to know the relative positions and orientations of each camera so that an accurate mapping or projection of the object position between each camera view can be performed.

Conventionally, the position and orientation of each camera is input manually by a technician during a calibration stage of installation of the monitoring system. This necessarily requires significant time and effort and the accuracy of the positions/orientations are limited by the technician's competence. Furthermore, each time a camera is to be repositioned or a new camera added to the monitoring system, the calibration procedure must be performed again at the cost of system downtime and technician's charges.

<CIT> relates to a stereo vision system for robotics, the system comprising cameras situated on a housing and positioned/oriented to view a scene in front of the system.

<CIT> relates to a surroundings monitoring device for heavy machinery in which a number of cameras are mounted on the machinery to view a surrounding scene.

<NPL>, relates to a system for determining a pose of 3D objects in depth data.

In accordance with a first aspect of the present invention there is provided a method of registering the position and orientation of one or more cameras in a camera imaging system according to claim <NUM>.

In some embodiments, step d) includes obtaining orientation data from orientation sensors mounted to the one or more cameras. In one embodiment, each orientation sensor includes an accelerometer. In one embodiment, each orientation sensor includes a gyroscope. In one embodiment, each orientation sensor includes a magnetometer. In one embodiment, each orientation sensor includes an inertial measurement unit.

In some embodiments, the depth imaging device includes one or more of a time of flight camera, structured light 3D scanner, stereo camera system, depth camera, laser scanner or LiDAR system.

The scene preferably includes a vehicle cabin or cockpit. Preferably, the scene is a vehicle and the reference orientation is defined relative to a vehicle axle or a vehicle drive shaft.

In some embodiments, the method includes the step of receiving velocity or acceleration data of the scene. In some embodiments, the method includes the step of receiving GPS data for a predefined position within the scene.

In some embodiments, the step of determining the three dimensional position of the cameras within the scene includes performing shape recognition on the three dimensional image to automatically recognize the position of the cameras in the image. In one embodiment, the shape recognition is adapted to recognize the shape of the cameras. In another embodiment, the shape recognition is adapted to recognize predefined patterns disposed on or adjacent each camera.

In some embodiments, the step of identifying the three dimensional position of the cameras within the scene includes manually designating the camera positions in the three dimensional image using a software application.

In one embodiment, the reference position is the position of one of the cameras. In another embodiment, the reference position is the position of a known feature within the scene.

In some embodiments, the step of determining a three dimensional position of each of the cameras within the scene includes calibrating the scene geometry with a known reference feature.

In one embodiment, the method includes the step:
f) removing the depth imaging device from the scene.

In accordance with a second aspect of the present invention, there is provided a system for registering the position and orientation of one or more cameras in a camera imaging according to claim <NUM>.

In some embodiments, the system includes one or more orientation sensors, each being mounted to an associated camera and adapted for measuring orientation data indicative of the orientation of the associated camera in at least one dimension.

Preferred embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:.

The embodiments of the present invention described herein relate to automating the registration of positions and orientations of cameras in a multi-camera vehicle monitoring system. In these embodiments, the scene to be imaged includes a driver of a vehicle, the interior of the vehicle/cockpit, the forward road scene of the vehicle and optionally side and rear views from the vehicle. The vehicle may represent a commercial automobile, truck, earthmoving machine, airplane, jet, helicopter or equivalent simulation environment. However, it will be appreciated that the invention is applicable to other multi-camera monitoring systems such as surveillance networks for monitoring other scenes. The invention may also be applicable to single camera monitoring systems.

Referring initially to <FIG> and <FIG>, there is illustrated a vehicle monitoring system <NUM> including four cameras <NUM>-<NUM> disposed at different locations within a vehicle <NUM>. Camera <NUM> is positioned on a dashboard of the vehicle and oriented in a forward direction of the vehicle for monitoring the forward road scene. Cameras <NUM>, <NUM> and <NUM> are positioned and oriented to monitor a driver <NUM> of vehicle <NUM>. Camera <NUM> is positioned on or adjacent an instrument panel of vehicle <NUM> or on the steering column of vehicle <NUM>. Camera <NUM> is positioned on the driver side A-pillar of the frame of vehicle <NUM>. Camera <NUM> is positioned on or adjacent a center console of vehicle <NUM>, preferably adjacent a display screen in vehicles where such a screen is provided. The specific camera locations are exemplary only and it will be appreciated that more or less cameras can be incorporated at other locations within or outside vehicle <NUM> to monitor the driver, the forward road scene or other views in or around the vehicle. Other exemplary locations of cameras include a rearview mirror, rear bumper, front bumper, vehicle roof and bonnet/hood.

Referring now to <FIG>, there is illustrated a system level diagram of system <NUM>. System <NUM> includes a central processing unit <NUM> including a processor <NUM>, memory <NUM>, a power source <NUM>, a network interface <NUM> and a user input device <NUM>. In the embodiments of a vehicle monitoring system, central processing unit <NUM> is preferably mounted within the vehicle dash or center console and can be integrated with an onboard vehicle computer system during manufacture. However, central processing unit <NUM> and system <NUM> as a whole may be manufactured as an after-market product and subsequently installed into vehicle in a modular manner.

Processor <NUM> may represent any known or conventional microprocessor or personal computer having hardware and/or software configured for processing image streams received from multiple cameras. By way of example, processor <NUM> may include system-on-chip technology and include a video processing pipeline. In one embodiment, processor <NUM> may be integral with or in communication with a processor of an onboard vehicle computer system.

Central processing unit <NUM> is powered by connection to a power source <NUM>. In one embodiment, power source <NUM> represents an electrical connection to a vehicle power source such as the vehicle battery. In another embodiment, power source <NUM> represents a local battery integrated within a housing of central processing unit <NUM> and optionally connected to an external power source.

Network interface <NUM> provides for communicating data to and from system <NUM> and represents an electrical or wireless interface for connecting system <NUM> to other devices or systems. Network interface <NUM> includes wired network ports such as USB, HDMI or Ethernet ports, serial device ports and/or wireless devices such as a Bluetooth™ device, Wi-Fi™ device or cellular network transceiver.

User input is able to be provided to system <NUM> through user input device <NUM>, which can include a touchscreen display or a keyboard or keypad and associated display. User input device <NUM> may also represent external devices such as computers or smartphones connected to system <NUM> through network interface <NUM> or other means. In one embodiment, user input device <NUM> represents a computer system integrated into the vehicle and manipulated through a display interface mounted in the vehicle's center console.

Example data that can be input to system <NUM> through user input device <NUM> includes:.

Example data that can be extracted from system <NUM> through user input device <NUM> includes:.

System <NUM> includes four camera units <NUM>-<NUM>, which are mounted at relative locations within or about the scene to be monitored. Each camera unit <NUM>-<NUM> includes a respective camera <NUM>-<NUM> for capturing images of the scene within its respective field of view.

Each camera is electrically connected to central processing unit <NUM> through respective connections <NUM>-<NUM> including electrical cables and associated electrical ports. The electrical connections provide for control of cameras <NUM>-<NUM> by processor <NUM> and transmission of image data from cameras <NUM>-<NUM>.

Cameras <NUM>-<NUM> may utilize various types of known image sensors in combination with imaging optics. Example image sensors include charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) chips combined with relevant processing electronics and memory to capture images and/or video sequences in suitable formats for subsequent image processing. Cameras <NUM>-<NUM> may be capable of capturing images in two or three dimensions.

In the case of a vehicle scene, the frame of reference may be defined relative to a region of the vehicle frame. By way of example, a reference coordinate system may be defined as having a z-axis aligned along the vehicle drive shaft (longitudinal dimension), an x-axis aligned along the front wheel axle (defining a transverse dimension) with the right wheel being in the positive direction and a y-axis defining a generally vertical dimension to complete the orthogonal coordinate system. This exemplary coordinate system will be used herein to describe the invention. However, it will be appreciated that other arbitrary reference coordinate systems may be chosen.

Finally, system <NUM> includes a depth imaging device <NUM> for capturing images of the scene, including each camera <NUM>-<NUM> in three dimensions. Depth imaging device <NUM> can include one or more of a scanning or pulsed time of flight camera, LiDAR system, stereoscopic camera arrangement, structured light 3D scanner, image sensor with phase detection or any other imaging system capable of capturing images of a scene in three dimensions. Depth imaging device <NUM> is operatively associated with processor <NUM> through electrical connection <NUM> to provide control to device <NUM> and receive raw three dimensional image data or pre-processed depth map data from device <NUM>. In some embodiments depth imaging device <NUM> is connected to central processing unit <NUM> and processor <NUM> through network interface <NUM>.

An alternative embodiment system <NUM> is illustrated in <FIG>. Here corresponding features of system <NUM> are designated with the same reference numerals. System <NUM> includes four camera units <NUM>-<NUM>, which each include a respective camera <NUM>-<NUM> and an orientation sensor <NUM>-<NUM> for measuring the orientation of the camera relative to a reference orientation.

Orientation sensors <NUM>-<NUM> may include simple inertial devices such as accelerometers and gyroscopes and other devices such as magnetometers and more advanced inertial measurement units, or combinations thereof. Orientation sensors <NUM>-<NUM> may be capable of measuring orientation in one, two or three dimensions relative to a reference orientation. A suitable reference orientation is that described above using the vehicle drive shaft and front wheel axle. However, it will be appreciated that a reference orientation can be chosen arbitrarily based on the particular application. For example, if two or more cameras were aligned along a common axis, that axis may be preferred as the reference orientation. The orientations are preferably expressed in a three dimensional Cartesian coordinate system. However, it will be appreciated that the orientations can be expressed in any arbitrary coordinate system such as a spherical coordinate system wherein an orientation vector is expressed in terms of a radial distance (r), a zenith angle (θ) in a vertical plane and an azimuthal angle (φ) in a horizontal plane.

In one embodiment, the orientation sensors <NUM>-<NUM> are mounted integrally on respective cameras <NUM>-<NUM>. In another embodiment, orientation sensors <NUM>-<NUM> are mounted relative to each camera <NUM>-<NUM> on an intermediate support frame on which the camera is also mounted.

The operation of systems <NUM> and <NUM> for registering the position and orientation of a plurality of cameras in a multi-camera imaging system will be described with reference to method <NUM> illustrated in the flow chart of <FIG>.

Initially, at step <NUM> the cameras are installed in their desired locations within the scene to be monitored. In the case of a driver monitoring system, the cameras are preferably mounted at locations in or around the vehicle such as those in <FIG> and <FIG> so as to position the driver and the forward road scene squarely within their respective fields of view. Step <NUM> may be performed during manufacture of the vehicle or during a subsequent installation of system <NUM> in vehicle <NUM>.

During the installation of the camera units <NUM>-<NUM>, each unit is electrically connected to central processing unit <NUM> through respective connections <NUM>-<NUM>. During step <NUM>, depth imaging device <NUM> is also installed and connected to system <NUM> through electrical connection <NUM>. Depth imaging device <NUM> may be installed in a temporary or permanent manner within or adjacent vehicle <NUM>.

Device <NUM> is installed with an accurately known camera pose relative to a fixed vehicle frame of reference defined in the next step. The camera pose includes a six dimensional vector indicating the three dimensional position of the camera sensor array and three dimensional orientation of an optical axis of the camera.

At step <NUM>, depth imaging device <NUM> is controlled to generate one or more three dimensional images of the scene, which includes camera units <NUM>-<NUM>. The three dimensional images represent depth maps of the scene including two dimensional transverse positions of objects plus a third dimension indicating the depth of objects from the depth imaging device <NUM>.

Prior to extracting accurate depth data, depth imaging device <NUM> will typically require a calibration. Some commercial devices such as time of flight cameras have inbuilt calibration routines which allow for self-calibration. However, other devices will require manual calibration by, for example, placing a calibration object in the scene which has dimensions known to a high degree of accuracy.

In the case where depth imaging device <NUM> includes a stereoscopic camera system, the two cameras first need to be calibrated by determining the relative position and orientation of the cameras relative to each other or a reference point to compute the ray intersection point for associated pixels of the cameras. Furthermore, there is a need to identify where each surface point that is visible in one stereo image is located in the other stereo image. The cameras must also be positioned so as to be capturing overlapping fields of view. Calibration of the scene can be initially performed by placing an object having a predefined calibration pattern in the scene to be imaged. Upon capturing stereo images, the calibration pattern can be used to identify corresponding pixels of the cameras and also to identify other parameters of the simulated 3D scene, such as the rotation and shift in three dimensions between the cameras, focal lengths, distortion etc. Using these parameters, the three dimensional coordinates of objects can be calculated within the scene.

The three dimensional image captured at step <NUM> is in the frame of reference of the depth imaging device <NUM>. At step <NUM>, the three dimensional image is calibrated to a reference frame within or relative to the vehicle scene to determine the pose of device <NUM> in the reference frame. The reference frame includes a reference position and reference orientation. A suitable reference frame is that described above using the vehicle drive shaft and front wheel axle. This reference frame will be referred to as the vehicle reference frame. Here reference position will be the intersection of the front vehicle axle and the drive shaft. The reference orientation may be designated as the forward direction of the vehicle defined along the longitudinal axis.

Other example reference positions include the three dimensional position of depth imaging device <NUM> or the three dimensional position of one of camera units <NUM>-<NUM>. However, the reference position can be chosen arbitrarily as long as it can be identified in the three dimensional images and accurately related to positions of the camera units <NUM>-<NUM>. By way of example, the reference position can be designated as a known point of reference within the scene such as a point on the center console of the vehicle. The reference position and orientation may be designated by a user through user input device <NUM>.

In some embodiments, step <NUM> may be performed manually by designating the position and orientation of device <NUM> through user input device <NUM>. In other embodiments, step <NUM> is performed through a self-calibration process <NUM> illustrated in <FIG> and described as follows. Initially, at step <NUM>, reference data indicative of the vehicle scene is loaded from memory <NUM>. The reference data includes positions and orientations of one or more known features within the vehicle scene. The reference data may represent a three dimensional CAD model of the vehicle scene or cabin. Alternatively, the reference data may represent an image captured by device <NUM> at an earlier time and at an accurately known camera pose. The known features may include features, objects, contours or surfaces within the vehicle scene such as dashboard instruments. The vehicle cabin itself may represent a known object if an accurate model of the cabin is known. The known objects must have a geometric appearance that is known within the scene.

Next, at step <NUM>, the geometric appearance of one or more of the known features is identified within the three dimensional image. This may occur through pattern matching, shape recognition or the like. Finally, at step <NUM>, the three dimensional position and orientation of the camera relative to the known features is determined from the geometric appearance. From this, a pose of the three dimensional camera within the vehicle scene can be determined. The final step can leverage use of a machine learning technique such as a convolutional neural network to learn the vehicle cabin and the features within it under different conditions. For example, when the driver is seated in the vehicle and occluding some of the known objects.

Returning to <FIG>, at step <NUM>, a three dimensional position of each of cameras <NUM>-<NUM> within the three dimensional image(s) in the reference frame is determined. In one embodiment, identifying the three dimensional position of the cameras within the scene includes manually designating the camera positions using a software application. In this technique, the three dimensional image is displayed on a display screen and a user is able to control a cursor through user input device <NUM> such as a mouse, keyboard, touchscreen or other similar device to manually designate the cameras using a select and click process. The selected positions are stored in memory <NUM> in a coordinate system defined with reference to the reference position.

In another embodiment, step <NUM> includes performing shape recognition on the three dimensional image to automatically recognize the position of cameras <NUM>-<NUM> in the image. If cameras <NUM>-<NUM> have a predefined and recognizable shape, a shape recognition algorithm may be performed by processor <NUM> to automatically recognize the two dimensional transverse position of each camera within the three dimensional image and subsequently extract the corresponding depth values of the cameras at those two transverse positions. If the resolution of the scene image is sufficiently high, a center of the camera aperture can be identified and designated as the camera position. As the cameras will generally comprise a region of pixels within the image, the depth of each camera can be calculated as an aggregate or average depth, or by a depth of a particular portion of the cameras.

As there is also a need to distinguish the cameras from one another, the automatic shape recognition algorithm described above may be used in conjunction with user input by user input device <NUM>.

In a further embodiment, each camera is designated with a predefined pattern prior to imaging and a shape recognition algorithm is performed which recognizes the predefined patterns to locate and distinguish each camera. Example patterns include a simple numerical identifier ('<NUM>', '<NUM>', '<NUM>'. ),a two or three dimensional barcode or other unique pattern. The patterns may be adhered to each camera using an adhesive material or may be scribed into a face of each camera. Preferably the pattern is located on a central region of the camera body so as to accurately represent the position of the camera image sensor. Further, the patterns may be printed on multiple faces of each camera to increase the chances of an accurate location in the image from different angles.

At step <NUM>, an orientation of each camera is determined in the vehicle frame of reference. In one embodiment utilizing system <NUM> described above, this includes obtaining orientation data of each camera from respective orientation sensors <NUM>-<NUM> through signals <NUM>-<NUM>. Preferably the orientation data includes a camera orientation in two or three dimensions. However, the orientation sensors may only provide orientation in one dimension and the remaining dimensions can be measured manually. The orientations are defined relative to the reference orientation described above. When utilizing orientation sensors to obtain the orientation of the cameras, step <NUM> can be performed in conjunction with steps <NUM> and <NUM>, or performed before or after steps <NUM> and <NUM>.

In an alternative embodiment, the orientation of cameras <NUM>-<NUM> can be determined through a self-calibration process similar to process <NUM> described above in relation to depth imaging device <NUM>. Here the scene camera under test is used to capture an image from its current pose and steps <NUM>-<NUM> are performed using the captured image. Essentially the captured image is compared to a three dimensional model of the scene (or an earlier captured image at a known camera pose) and pattern matching and/or machine learning is used to determine the orientation of the camera within the scene in the vehicle reference frame. This process can be performed for each camera and can also be used to capture camera position or as a check for the accuracy of the above measurement in step <NUM>. Thus, this process could be used to perform both steps <NUM> and <NUM>.

At optional step <NUM>, processor <NUM> receives external input such as velocity or acceleration data of the vehicle, or a GPS position of the vehicle. By way of example, system <NUM> may be connected to an onboard vehicle computer system through network interface <NUM> which provides a feed of vehicle data such as vehicle GPS position, velocity and acceleration. The external input data can be used to augment the orientation data to aid in the calculation of camera orientations, particularly when the orientation sensor is only capable of measuring orientation in one or two dimensions. By way of example, yaw orientation (in the vertical axis) is typically difficult to determine when the camera is stationary under the constant force of gravity. Measuring a change in position, velocity or acceleration of the camera as the vehicle is in motion can help to determine an overall force vector from which the orientation of the camera in three dimensions can be extrapolated.

The external input can also be fed to a machine learning algorithm as additional inputs for the determination of camera positions in steps <NUM> and <NUM>.

Finally, at step <NUM>, processor <NUM> combines the three dimensional position, orientation and optionally the external input data for each camera to determine a camera pose.

The camera pose can be used to identify and track objects within the scene across different cameras. In one embodiment, a driver's gaze point (point of regard) can be deduced from a driver facing camera and projected onto images of a forward facing camera capturing the forward road scene.

It will be appreciated that the system and method described above provides for efficiently and accurately determining relative positions and orientations of cameras in a multi-camera system.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "processing," "computing," "calculating," "determining", analyzing" or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing device, that manipulate and/or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

In a similar manner, the term "controller" or "processor" may refer to any device or portion of a device that processes electronic data, e.g., from registers and/or memory to transform that electronic data into other electronic data that, e.g., may be stored in registers and/or memory.

The methodologies described herein are, in one embodiment, performable by one or more processors that accept computer-readable (also called machine-readable) code containing a set of instructions that when executed by one or more of the processors carry out at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken are included. Thus, one example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system further may include a memory subsystem including main RAM and/or a static RAM, and/or ROM. A bus subsystem may be included for communicating between the components. The processing system further may be a distributed processing system with processors coupled by a network. If the processing system requires a display, such a display may be included, e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and so forth. The term memory unit as used herein, if clear from the context and unless explicitly stated otherwise, also encompasses a storage system such as a disk drive unit. The processing system in some configurations may include a sound output device, and a network interface device. The memory subsystem thus includes a computer-readable carrier medium that carries computer-readable code (e.g., software) including a set of instructions to cause performing, when executed by one or more processors, one of more of the methods described herein. Note that when the method includes several elements, e.g., several steps, no ordering of such elements is implied, unless specifically stated. The software may reside in the hard disk, or may also reside, completely or at least partially, within the RAM and/or within the processor during execution thereof by the computer system. Thus, the memory and the processor also constitute computer-readable carrier medium carrying computer-readable code.

Furthermore, a computer-readable carrier medium may form, or be included in a computer program product.

In alternative embodiments, the one or more processors operate as a standalone device or may be connected, e.g., networked to other processor(s), in a networked deployment, the one or more processors may operate in the capacity of a server or a user machine in server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.

Note that while diagrams only show a single processor and a single memory that carries the computer-readable code, those in the art will understand that many of the components described above are included, but not explicitly shown or described in order not to obscure the inventive aspect. For example, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

Thus, one embodiment of each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, e.g., a computer program that is for execution on one or more processors, e.g., one or more processors that are part of web server arrangement. Thus, as will be appreciated by those skilled in the art, embodiments of the present invention may be embodied as a method, an apparatus such as a special purpose apparatus, an apparatus such as a data processing system, or a computer-readable carrier medium, e.g., a computer program product. The computer-readable carrier medium carries computer readable code including a set of instructions that when executed on one or more processors cause the processor or processors to implement a method. Accordingly, aspects of the present invention may take the form of a method, an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.

The software may further be transmitted or received over a network via a network interface device. While the carrier medium is shown in an exemplary embodiment to be a single medium, the term "carrier medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term "carrier medium" shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by one or more of the processors and that cause the one or more processors to perform any one or more of the methodologies of the present invention. A carrier medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical, magnetic disks, and magneto-optical disks. Volatile media includes dynamic memory, such as main memory. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise a bus subsystem. Transmission media also may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. For example, the term "carrier medium" shall accordingly be taken to included, but not be limited to, solid-state memories, a computer product embodied in optical and magnetic media; a medium bearing a propagated signal detectable by at least one processor of one or more processors and representing a set of instructions that, when executed, implement a method; and a transmission medium in a network bearing a propagated signal detectable by at least one processor of the one or more processors and representing the set of instructions.

It will be understood that the steps of methods discussed are performed in one embodiment by an appropriate processor (or processors) of a processing (i.e., computer) system executing instructions (computer-readable code) stored in storage. It will also be understood that the invention is not limited to any particular implementation or programming technique and that the invention may be implemented using any appropriate techniques for implementing the functionality described herein. The invention is not limited to any particular programming language or operating system.

Reference throughout this specification to "one embodiment", "some embodiments" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases "in one embodiment", "in some embodiments" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment.

As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

It should be appreciated that in the above description of exemplary embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single embodiment, Figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this disclosure.

Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the disclosure, and form different embodiments, as would be understood by those skilled in the art.

However, it is understood that embodiments of the disclosure may be practiced without these specific details.

Similarly, it is to be noticed that the term coupled, when used in the claims, should not be interpreted as being limited to direct connections only. The terms "coupled" and "connected," along with their derivatives, may be used. Thus, the scope of the expression a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means. "Coupled" may mean that two or more elements are either in direct physical, electrical or optical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.

Claim 1:
A computer implemented method of determining the position and orientation, relative to a vehicle reference frame, of one or more cameras (<NUM>, <NUM>, <NUM>, <NUM>) in a camera imaging system located within a vehicle, the method including:
a) receiving, from a depth imaging device (<NUM>), data indicative of a three dimensional image of a scene including the one or more cameras (<NUM>, <NUM>, <NUM>, <NUM>) located in the scene, the scene including an interior of a cabin or a cockpit of the vehicle;
b) calibrating (<NUM>) the three dimensional image with the vehicle reference frame relative to the scene, the reference frame including a reference position and a reference orientation relative to the vehicle in the vehicle frame of reference;
c) determining (<NUM>) a three dimensional position of each of the one or more cameras (<NUM>, <NUM>, <NUM>, <NUM>) within the three dimensional image in the vehicle reference frame by determination of the geometric appearance of the one or more cameras in the three dimensional image;
d) determining (<NUM>) an orientation of each of the one or more cameras (<NUM>, <NUM>, <NUM>, <NUM>) in at least one dimension in the vehicle reference frame; and
e) combining (<NUM>) the position and orientation of each of the one or more cameras (<NUM>, <NUM>, <NUM>, <NUM>) to determine a camera pose for each of the one or more cameras in the vehicle reference frame.