Motion capture system

Systems and methods for motion capture are disclosed. In one arrangement a plurality of cameras are provided and a processing unit is configured to perform processing using image data from each of the plurality of cameras. Each of the cameras comprises an accelerometer and the processing unit is configured to perform processing that is dependent on an output from the accelerometers. In another arrangement each of the cameras comprises a temperature sensor and the processing unit is configured to perform processing that is dependent on an output from the temperature sensors.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a U.S. national stage application of International Patent Application No. PCT/GB2016/051130, filed Apr. 22, 2016, which claims the benefit of priority to GB Patent Application No. 1509993.0, filed Jun. 9, 2015, the entireties of which are incorporated herein by reference.

The present invention relates to a motion capture system and a method of performing motion capture, in which a plurality of cameras are each provided with an accelerometer and/or a temperature sensor.

A motion capture system typically comprises a plurality of cameras configured to monitor a target region in which the movements of a target, for example an actor, are to be captured. The captured movements may be used to animate a digital character in a film or computer game based on the actor's movements, for example. Typically, in such systems each of the cameras will need to be mounted at separate locations and directed towards the target region. The cameras will also need to be calibrated and connected to a processing unit such as a desktop or laptop computer, or a tablet. Setting up such motion capture systems can be time consuming, requiring for example multiple return trips for an operator between the processing unit and each of the cameras and/or the involvement of multiple people. For example, one person might interact directly with the processing unit while one or more other people may move around the room individually adjusting and checking each of the cameras while communicating with the person at the processing unit. The person at the processing unit may provide feedback to the moving operator about the image being captured by each of the cameras, as the moving operator moves from one camera to the next.

Motion capture systems may be sensitive to environmental conditions and to disturbances. For example, deviations in the quality of a camera mounting (e.g. due to knocks, thermal expansion/contraction, bending, etc.) may cause the position or orientation of the camera to change over time. This can have a negative impact on the motion capture process and/or require frequent and time consuming recalibrations.

A further challenge with existing systems is that temperature variations within the cameras themselves may alter the performance of the cameras, for example by causing differences in the optical path from a lens through to an imager of the camera. These effects can be mitigated in some circumstances by calibrating the cameras after the temperature has reached a steady state. However, judging when to perform such calibration is difficult. Users may be advised to wait a predetermined period after start up of the system before performing camera calibration and beginning use of the system. However, this can cause inconvenient delay. Furthermore, such an approach is not effective where a steady state temperature is not reached reproducibly or where the temperature varies during use of the system, for example due to variations in the activity of individual cameras or in the temperature of the surrounding environment.

It is an object of the invention to provide a motion capture system and method that at least partially address one or more of the challenges discussed above.

According to an aspect of the invention, there is provided a motion capture system comprising: a plurality of cameras; and a processing unit configured to perform processing using image data from each of the plurality of cameras, wherein each of the cameras comprises an accelerometer and the processing unit is configured to perform processing that is dependent on an output from the accelerometers.

Thus, a system is provided in which individual cameras can respond to output from an accelerometer provided in each camera. This capability enhances flexibility in how the motion capture system can be set up and maintained, facilitating in particular procedures which involve close proximity of a user with individual cameras. For example, the system can be configured to use the accelerometer outputs to allow a user to select individual cameras by causing perturbations in the accelerometer outputs, e.g. by tapping or double tapping a camera. The accelerometers may alternatively or additionally be used to detect unwanted deviations in the orientation of cameras, thereby to prompt investigatory action and/or recalibration procedures.

According to an aspect of the invention, there is provided a motion capture system comprising: a plurality of cameras; and a processing unit configured to perform processing using image data from each of the plurality of cameras, wherein each of the cameras comprises a temperature sensor and the processing unit is configured to perform processing that is dependent on an output from the temperature sensors.

Thus, a system is provided in which individual cameras can respond to variations in a temperature of each camera. This capability facilitates detection of when a steady state temperature has been reached after a powering on of the camera, making it possible for calibration procedures to be started more reliably in the steady state regime, and potentially earlier than would be possible using other approaches. The capability also makes it possible to respond to variations in temperature that may occur at times other than shortly after powering on, for example due to changes in the environmental temperature.

According to an alternative aspect of the invention, there is provided a method of performing motion capture using a plurality of cameras, wherein each of the cameras comprises an accelerometer, the method comprising: performing processing at a processing unit that is dependent on an output from the accelerometers.

According to an alternative aspect of the invention, there is provided a method of performing motion capture using a plurality of cameras, wherein each of the cameras comprises a temperature sensor, the method comprising: performing processing at a processing unit that is dependent on an output from the temperature sensor.

In an embodiment, an example of which is shown inFIGS. 1-3, there is provided a motion capture system2comprising a plurality of cameras4. A processing unit6is configured to perform processing using image data received from each of the plurality of cameras4. The processing may comprise processing associated with performing motion capture for example. In this context, motion capture is understood to encompass at least any process or technique of recording patterns of movement digitally, for example for recording of an actor's movements for the purpose of animating a digital character in a film or computer game. Optionally, one or more of the plurality of cameras4may be configured to perform processing that contributes to the motion capture procedure. For example a camera may pre-process image data captured by the camera before sending data to the processing unit. The processing unit6may take various forms. For example, the processing unit6may comprise a computer (e.g. a desktop or laptop PC), a tablet computer, a smart telephone, or any combination of these in communication with each other. The processing unit6may comprise a display8. The processing unit6may further comprise processing hardware10(e.g. CPU, memory, etc.), and an interface12allowing a user to interact with the processing unit6. The interface may comprise a touch screen, keyboard, mouse, or other input device.

In an embodiment each of the cameras4comprises an accelerometer22. An example of such a camera is shown inFIG. 3. In embodiments of this type the processing unit6is configured to perform processing that is dependent on an output from the accelerometers22.

In an embodiment the processing unit6is configured to select one of the cameras4based on an output from the accelerometer22of that camera. The processing unit6is further configured to perform a selected-camera processing which depends on which of the cameras4has been selected. The selected-camera processing may comprise various different functionalities associated with the selected camera. In an embodiment, the selected-camera processing comprises displaying data, for example on the display8, from the selected camera4. This functionality therefore makes it possible for a user to view data associated with a particular camera merely by causing an output from the accelerometer to vary in a particular way (e.g. by tapping or otherwise manipulating the camera4).

In an embodiment, the selected-camera processing comprises displaying data from a plurality of the cameras and providing visual information indicating which of the displayed data corresponds to the selected camera. For example, the processing unit6may be configured to display image data captured by more than one of the cameras4in a corresponding plurality of different windows in the display8. The selected camera may be indicated by highlighting the window corresponding to the selected camera, or by bringing the window to the front relative to other windows, for example.

Alternatively or additionally, the selected-camera processing may comprise displaying data from the selected camera and not displaying data from any other camera. In this way, manipulation of the camera to cause a characteristic change in the output from the accelerometer22in that camera can be used to cause the display8to switch between displaying data from different cameras.

The above-described functionality may be particularly convenient in the case where the processing unit6comprises a portable unit, such as a laptop or tablet. In this case, a user can carry the portable unit from one camera4to another camera4in the system and conveniently select to view data associated with a camera that he is currently next to merely by causing the output from the accelerometer to change (e.g. by tapping the camera4). This functionality avoids the user having to manually interact with the portable unit itself in order to select a camera. This approach may save considerable time and effort, particularly in the case where many different cameras4are provided.

In an embodiment, the displayed data of the selected-camera processing comprises image data captured by the selected camera4. Alternatively or additionally, the displayed data comprises configuration data of the selected camera. The configuration data may comprise any operating parameters or other parameters of the camera4. In an embodiment, the selected-camera processing comprises providing a selected camera interface, for example as part of the display8, for example a touch screen active part of the display12, which allows a user to control the selected camera4. Thus, by tapping or otherwise manipulating the camera4to provide a change in the output from the accelerometer22, the user is able conveniently to bring up an interface which allows him to control the selected camera4via the processing unit6(e.g. laptop or tablet).

The selected-camera interface may allow control of a plurality of cameras4and provide a visual indication indicating which of the cameras4is the selected camera4. This approach allows the user to view configuration data or other relevant data from other cameras while controlling the selected camera4. This may be useful where the data from other cameras is relevant for comparison or reference purposes.

In an embodiment, the selected-camera interface is configured to allow control of the selected camera4and no other camera4. Various other functionalities associated with the selected camera4may be provided by the selected-camera interface.

In an embodiment, the processing unit6is configured to select one of the cameras4in response to a perturbation in the output of the accelerometer22(e.g. an amplitude of the perturbation, proportional to a magnitude of an acceleration, or a duration of the perturbation) from that camera4that is larger than a predetermined threshold. The perturbation may be caused for example by a user of the system tapping the camera4lightly.

Alternatively or additionally, the processing unit6may be configured to select one of the cameras4when an output from the accelerometer22from that camera4matches a predetermined signature output. The predetermined signature output may correspond to an output consistent with a user tapping the camera with a finger, or any other manipulation of the camera by a user which does not disturb the mounting of the camera4. Typically, therefore, the processing unit6is configured to detect relatively light interaction between the user and the camera4, such as tapping (single or multiple, such as double tapping), or other similar, light interactions. The predetermined signature output may be determined by recording the response of the camera accelerometer22to one or more types of envisaged interaction. For example, the response to a variety of single finger taps could be recorded and an average response could be used as a predetermined signature output. When a user taps the camera4in use, the response of the accelerometer22can be compared to the recorded average response and if the two responses are sufficiently similar it can be deduced that there is a match between the output from the accelerometer and the predetermined signature output. Various aspects of the accelerometer output could be used for the comparison, including for example the duration of the perturbation in the output, the amplitude of the perturbation of the output, or the variation with time of the amplitude of the perturbation. Using the variation with time of the amplitude of the perturbation may be particularly appropriate for example where the interaction by the user takes a more complex form, for example a double tap. Here, it may be appropriate for example to detect a pair of local maxima in the perturbation of the accelerator output22and/or the time separation of the maxima in order to determine whether the user interaction matches the predetermined signature output for a double tap.

In an embodiment, the processing unit6is configured to respond differently according to the direction of a perturbation in the output from the accelerometer22. For example, the processing unit6may be configured to perform a first direction-selected processing when the camera4is tapped on one side (and thus in a first direction, for example in an inward direction substantially perpendicular to the first side) and a second direction-selected processing, different from the first direction-selected processing, when the camera4is tapped on a second side (and thus in a second direction, different from the first direction, for example in an inward direction substantially perpendicular to the second side). For example, the first direction-selected processing may comprise displaying image data captured by the camera and the second direction-selected processing may comprise displaying configuration data of the camera.

FIG. 2depicts how four cameras4of a motion capture system2may be configured so as to be directed towards a target region16. The provision of four cameras4is exemplary. Fewer than four or more than four cameras4may be provided according to requirements. Typically, each of the cameras4will be a physically separate unit. Each camera4is typically mountable independently of the other cameras4. In the example shown, the cameras2are mounted to separate mounting supports14.

The cameras4may be configured to interact with the processing unit6in various ways. Wires may be provided between each of the cameras4and the processing unit6, for example. Alternatively or additionally, wireless data transfer may be possible between the cameras4and the processing unit6.

In the above-described embodiments, accelerometers22in the cameras4are used to provide a convenient way for a user to select a camera that he is in close proximity with and/or to select functionality associated with that camera. However, the accelerometers22may be used for other purposes, in addition to the functionalities described above, or as an alternative to the functionalities described above.

For example, in an embodiment the processing unit6is configured to detect a change in an orientation of one or more of the cameras4based on an output from the accelerometers22. This may be achieved by detecting a change in the direction of gravity relative to the camera4. This is illustrated schematically in the side view ofFIG. 3. Arrow26indicates the direction of gravity relative to the accelerometer22. Gravity acts as an acceleration and is therefore detected by the accelerometer22. The accelerometer22is able to detect the direction of gravity26relative to a reference axis28of the camera4. In the example shown inFIG. 3, this is used to allow the accelerometer22to determine an angle24of inclination of the camera4relative to the vertical. Typically, when the motion capture system2is set up, one or more of the cameras4will be mounted fixedly and then calibrated before the motion capture system is used to capture motion in the target region16. Any deviation in the position or orientation of a camera4after the calibration process has finished will lead to a reduction in the quality of the image capture process. Such deviations may occur for example due to knocks to the camera4or mounting14, variations in temperature and associated thermal expansion/contraction in the camera4or mounting14, or yielding or bending of the mounting14, etc. The inventors have recognised that a wide range of such deviations will involve a change in orientation of the camera4relative to the direction of gravity and that this can be detected by an accelerometer22. Thus, the accelerometer22provides a convenient and accurate way for the processing unit6to detect when there has been a deviations in the camera mounting. Even changes which occur slowly over time, such as when a mounting is gradually bending or changing due to long time-scale temperature variations, etc., the accelerometer22can still detect changes in the orientation by comparing an absolute measurement of the orientation at a camera4with an absolute recording of the orientation of the same camera4at a previous time. For example, when the system is left overnight, a comparison can be made between the outputs from the accelerometers22in the morning compared with the outputs recorded from the previous day. Any significant variations (e.g. variations that are larger than a predetermined threshold) will indicate that a camera has been disturbed significantly and that checking of the mounting and/or recalibration is therefore necessary.

In the above-described embodiments, each of the cameras4is provided with an accelerometer22. In these and other embodiments each of the cameras4may alternatively or additionally comprise one or more other sensors which provide useful information about the state of the camera.

In an embodiment, each of the cameras4comprises a temperature sensor18. The temperature sensor18may be provided instead of or in addition to the accelerometer22. The processing unit6is configured in such embodiments to perform processing that is dependent on an output from the temperature sensor18. In an embodiment the processing unit6is configured to provide a visual indication of the temperature of one or more of the cameras4. The visual indication may be provided on the display12of the processing unit6for example. Alternatively or additionally, a visual indication of the temperature may be provided at the individual cameras4. In an embodiment, the temperature sensor18of each camera4is mounted on computer processing hardware20, such as a PCB, within the camera4. The temperature sensor18in such an embodiment is therefore configured to measure the temperature of the computer processing hardware20. Typically, when a camera4is powered on power is supplied to the computer processing hardware20within the camera4, which causes the hardware20to heat up. The heating can lead to changes in operating characteristics of the computer processing hardware20and/or in characteristics of other elements of the camera. For example, heat can be conducted from the computer processing hardware20to optical or mechanical elements of the camera, for example between a lens of the camera4and an image capture unit configured to transform light received at pixels thereof into an electrical signal representing the image captured. By measuring the temperature of the computer processing hardware20it is possible to detect for example when the temperature of the hardware20reaches a steady state. At this point, further thermal-driven changes in the properties of the camera4are likely to be very small. A user can therefore safely commence calibration procedures at this point in the knowledge that subsequent temperature variations are unlikely to cause significant deviation from the calibrated state. Relative to alternative approaches in which a user simply waits for a predetermined type which is chosen so as to be safely longer than any temperature settling period, the approach of the present embodiment allows a user to start the calibration procedure earlier and/or at a point which is more reliably in the steady state temperature regime.