Patent Description:
Numerous systems exist to monitor the positions of target objects in 3D space, including so-called co-ordinate measuring machines, laser trackers, photogrammetry and motion capture systems.

Some photogrammetry systems comprise a stereoscopic (stereo) camera assembly, in which two cameras are mounted a known distance apart by a rigid structure. The two cameras comprise respective lens assemblies associated with image sensors which are operable to capture a pair of stereoscopic images of a scene. The pair of stereoscopic images are then analysed using conventional photogrammetry techniques to determine 3D positions of targets within the scene, e.g. based on the relative displacement of the targets in the pair of stereoscopic images.

Such photogrammetry systems were originally designed to operate in well-controlled laboratory environments. However, the value of using such systems to perform 3D measurements in harsher environments, such as those encountered in shop-floor manufacturing settings, is recognised and some systems have been adapted with shop-floor applications in mind. For example, it is known to direct a single stereo camera assembly towards a manufacturing shopfloor scene that includes a target of interest, e.g. a robot end-effector, and to determine the relative position of the target from the stereo camera assembly based on an analysis of the stereoscopic images captured by the stereo camera assembly.

Despite prior efforts in the field, the above systems suffer a number of disadvantages that make it difficult to obtain position data that is accurate enough to be used in harsher or complex settings.

Firstly, the accuracy of a single stereo camera assembly is limited by its "base length", which is the separation distance between the lens assemblies. For a given base length, measurement errors tend to increase in a squared relationship with range, i.e. the distance of the target being monitored from the cameras; doubling the measurement range introduces aproximaterly four times the error. It may be possible to increase the accuracy of the cameras at long ranges (such as those typically encountered in a manufacturing setting) by increasing the base length. However, doing so is often impractical and itself introduces errors into the measurement because the rigid structure between the cameras will be susceptible to larger thermal and vibrational influences (e.g. when used in a manufacturing setting).

It is also known that, in complex environments, the line-of-sight of the cameras towards the target may be obstructed at times, thereby making it difficult to capture images of the target. For example, a target in the form of a robot end-effector may face (and thus be visible to) the cameras at the start of a manufacturing operation, but then turn to face away from the cameras during the operation, thereby interrupting relevant data capture. Manufacturing processes etc. often require continuous and real-time monitoring of target positions within the scene, such that conventional systems are unsuitable for such applications.

It is desired to provide a photogrammetry system that enables accurate and continued position monitoring of target objects so that it may be particularly suitable for use in complex settings.

<CIT> discloses a method and system for tracking objects, such as humans, over a wide area by processing images captured by each of a plurality or network of slightly overlapping stereo sensors, such as stereo cameras. <CIT> discloses an entity-tracking computing system that receives sensor information from a plurality of different sensors. The positions of entities detected by the various sensors are resolved to an environment-relative coordinate system so that entities identified by one sensor can be tracked across the fields of detection of other sensors. <CIT> discloses a device comprising: a shape measuring section for measuring shape data from a pair of stereovision images of an object taken with an image taking section; a memory section for storing target data related to the images of the object; an image display section for superposing the paired stereovision images of the object and the stereovision target data image based on the target data and for displaying the images; and a comparison display processing section for comparing the shape data of the object measured by the shape measuring section, with the target data and reflecting the results of the comparison on the superposed display on the image display section.

According to an aspect, there is provided a method according to claim <NUM>. According to a further aspect of the invention, there is provided a photogrammetry system according to claim <NUM>.

By determining a global position of the target based on the three-dimensional positions reported by (received from) plural nodes, the accuracy of the global positioning determination may be increased as compared to that achieved by conventional systems in which a single node or stereo camera assembly is used.

Furthermore, by determining three-dimensional positions of the target locally at each node and transmitting the three-dimensional positions (e.g. in the form of co-ordinate values) to the hub controller for further processing, the system is able to avoid data transfer bottlenecks that would otherwise occur if, for example, the image data itself is transferred from each node to the hub controller for central image processing. This may be advantageous in that it increases the processing efficiency of the system and therefore reduces the overall processing time, thereby allowing real-time position monitoring.

The two or more nodes may be located at different positions about the scene to capture images of the scene from unique viewpoints. This may reduce or even avoid the risk that line of sight to the target will be completely lost.

The system may be extensible in that the number of nodes in the network may be adjustable to include any number of nodes. In that regard, the number of nodes may be adjusted without substantially affecting operation of the nodes or the hub controller. For example, the hub controller may be configured to determine a global position of the target based on all of the three-dimensional positions received for that target (for a given time stamp) from the network of nodes, regardless of the number of nodes.

The network may include at least four nodes. This may achieve a minimum desired level of accuracy and versatility for monitoring a target within a complex setting.

The three-dimensional position included in the target data may be a relative position of the target from the node. The three-dimensional position may be represented in the form of three-dimensional co-ordinate values.

The hub controller may be configured to convert the relative position of the target to an absolute position in the first co-ordinate space, by determining a positional relationship between the target and a reference point. A position of the reference point within the first co-ordinate space may be known to the hub controller. The reference point may be a reference point on the node in question or a reference point corresponding to a reference target located within the scene.

A reference target may be located at the reference point within the scene. The processor may be configured to process the set of stereoscopic images to determine a three-dimensional position of the reference target relative to the node and to cause the node to transmit target data including the three-dimensional position of the reference target to the hub controller. The hub controller may be configured to determine the positional relationship between the target and the reference point by comparing their relative positions from the node.

The hub controller may be configured to receive reference data indicating the position of the reference point within the first co-ordinate space.

Each node may be configured to process the set of stereoscopic images to identify the target, if visible, and generate metadata indicating the determined identity of the target. The target data transmitted to the hub controller may further comprise the metadata. Metadata indicating the determined identity of the target may be advantageous to allow the hub controller to group three-dimensional positions corresponding to the same target. This in turn may allow the positions of multiple targets within the scene to be monitored.

The hub controller may be configured to issue a trigger command to the network of two or more nodes to control synchronous capture of respective sets of stereoscopic images.

The hub controller is configured to determine the global position of the target by: performing a best-fit operation on plural three-dimensional positions determined for the target.

The system may further comprise a flash unit for illuminating the scene with infrared radiation. The digital cameras may be sensitive to the infrared radiation such that the stereoscopic images represent the intensity of infrared radiation reflected by the scene.

The target may comprise a marker attached to an object within the scene.

The target may comprise a platform and a marker arranged at a first predefined position on the platform. The processor may be further configured to process the set of stereoscopic images to determine the position of the marker on the platform and identify the target based on the determined position.

The platform may have a plurality of attachment points arranged in a predefined array of positions on the platform. The marker may be suitable for being removably attached to the platform at any one of the attachment points at a time.

The target may comprise a plurality of markers arranged in a predefined pattern. The processor or the hub controller may be configured to determine an orientation of the target based on the three-dimensional positions of the plurality of markers.

Each marker may comprise a retroreflective material and/or may have a spherical shape.

Where the three-dimensional position included in the target data is a relative position of the target from the node, the method may comprise the hub controller converting the relative position of the target to an absolute position in the first co-ordinate space, by determining a positional relationship between the target and a reference point, wherein a position of the reference point within the first co-ordinate space is known to the hub controller.

A reference target may be located at the reference point within the scene. In such embodiments, the method may comprise the processor processing the set of stereoscopic images to determine a three-dimensional position of the reference target relative to the node and causing the node to transmit target data including the three-dimensional position of the reference target to the hub controller. The method may comprise the hub controller determining the positional relationship between the target and the reference point by comparing their relative positions from the node.

The method may comprise receiving, at the hub controller, reference data indicating the position of the reference point within the first co-ordinate space.

The method may comprise each node processing the set of stereoscopic images to identify the target within the scene and generating metadata indicating the determined identity of the target, wherein the target data further comprises the metadata.

The method may comprise the hub controller issuing a trigger command to instruct the nodes in the network to perform synchronous image capture. The method may comprise each node, in response to receiving the trigger command from the hub controller, capturing the set of stereoscopic images in synchronicity with the other nodes. Determining the global position of the target comprises performing a best-fit operation on plural three-dimensional positions determined for the target.

The best-fit operation is a weighted fit of the three-dimensional positions, wherein the weighting attributed to each three-dimensional position is based on the relative position of the target from the node to which the three-dimensional position corresponds. The method may comprise the two or more nodes processing the sets of stereoscopic images in parallel.

According to an aspect of the technology described herein, there is provided a target for use with a photogrammetry system, such as the system described above in the preceding statements. The target may have any one or more of the features described herein. Accordingly, the target may comprise a platform and a marker arranged at a first predefined position on the platform. The platform may have a plurality of attachment points arranged in a predefined array of positions on the platform. The marker may be suitable for being removably attached to the platform at any one of the attachment points.

The processor(s) and controller(s) (and various associated elements) described herein may comprise any suitable circuitry to cause performance of the methods described herein and as illustrated in the Figures. The processor or controller may comprise: at least one application specific integrated circuit (ASIC); and/or at least one field programmable gate array (FPGA); and/or single or multi-processor architectures; and/or sequential (Von Neumann)/parallel architectures; and/or at least one programmable logic controllers (PLCs); and/or at least one microprocessor; and/or at least one microcontroller; and/or a central processing unit (CPU), to perform the methods.

The processor or controller may include at least one microprocessor and may comprise a single core processor, may comprise multiple processor cores (such as a dual core processor or a quad core processor), or may comprise a plurality of processors (at least one of which may comprise multiple processor cores).

The processor or controller may be part of a system that includes an electronic display, which may be any suitable device for conveying information, e.g. position data, to a user.

The processor or controller may comprise and/or be in communication with one or more memories that store the data described herein, and/or that store software for performing the processes described herein.

The memory may be any suitable non-transitory computer readable storage medium, data storage device or devices, and may comprise a hard disk and/or solid state memory (such as flash memory). The memory may be permanent non-removable memory, or may be removable memory (such as a universal serial bus (USB) flash drive).

The memory may store a computer program comprising computer readable instructions that, when read by a processor or controller, causes performance of the methods described herein, and as illustrated in the Figures. The computer program may be software or firmware, or may be a combination of software and firmware.

The computer readable storage medium may be, for example, a USB flash drive, a compact disc (CD), a digital versatile disc (DVD) or a Blu-ray disc. In some examples, the computer readable instructions may be transferred to the memory via a wireless signal or via a wired signal.

It will be appreciated that like reference numerals are used in the drawings to label like features of the technology described herein.

<FIG> and <FIG> show an example of a photogrammetry system <NUM> which is suitable for determining and monitoring, in real-time, the position of target objects (referred to hereafter as "targets") in accordance with the technology described herein. In the example of <FIG> and <FIG>, the system is employed to monitor targets in a scene corresponding to a manufacturing setting, although it will be appreciated that the system <NUM> is applicable more generally to any type of setting. For example, the system could be employed in a medical operating theatre to monitor the position of a surgical tool.

The system <NUM> comprises an interconnected network of node devices <NUM>, <NUM>, <NUM>, <NUM>, <NUM> that are in wired communication with a hub controller <NUM> via one or more communications lines <NUM> (although wireless communication is equally possible). The hub controller <NUM> is operable using processing circuitry to send control signalling to the nodes <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, to control operation of the nodes in accordance with the methods described herein.

As best shown in <FIG>, which schematically illustrates one such node <NUM> of the system <NUM>, each node comprises a housing <NUM> which encloses a first digital camera <NUM> and a second digital camera <NUM>, which are mounted apart on a rigid support structure <NUM>. The cameras <NUM>, <NUM> and the associated support structure <NUM> form a stereo camera assembly.

Each camera <NUM>, <NUM> has an image sensor (not shown), e.g. a charge-coupled device (CCD), and an associated lens assembly <NUM>, <NUM>. The lens assemblies <NUM>, <NUM> are exposed on a common, exterior surface <NUM> of the housing <NUM> so as to direct external light (electromagnetic radiation) incident on the lens assemblies <NUM>, <NUM> towards their associated image sensors. The node <NUM> may also comprise at least one flash unit (not shown), e.g. one flash unit per camera <NUM>, <NUM>, which is operable to illuminate the scene to be imaged.

The first and second cameras <NUM>, <NUM>, and thus their corresponding lens assemblies <NUM>, <NUM>, are spaced apart by a known distance referred to herein as the base length <NUM> of the stereo camera assembly. In the system of <FIG> and <FIG>, the camera assembly in each node <NUM>, <NUM>, <NUM>, <NUM>, <NUM> has the same base length <NUM>, e.g. of <NUM>, although in other arrangements the base length <NUM> of the camera assembly may differ from node-to-node.

The node <NUM> further comprises an on-board processor <NUM>, which is operable (e.g. under the direction of the hub controller <NUM>) to cause the cameras <NUM>, <NUM> to simultaneously capture images representing the light incident on the sensors. Each stereo camera assembly will generate a set (in this example a pair) of stereo images representing the scene falling within the fields of view of the lens assemblies <NUM>, <NUM>. In that regard, the lens assemblies <NUM>, <NUM> are oriented such that their fields of view at least partly overlap. In this way, any target located within the overlapping portion will be represented in both stereo images, albeit at offset positions in each image due to the base length spacing between the lens assemblies <NUM>, <NUM>.

The on-board processor <NUM> is configured to perform edge processing at the node, including processing of the stereo images generated by the cameras <NUM>, <NUM> in the node <NUM>. As will be described in further detail below, the processor <NUM> is configured to process the pair of stereo images to determine 3D positions of targets within the scene. To facilitate this, the processor <NUM> may receive, or otherwise be provided with, all information necessary to determine the positions of the targets based on the image data, such as the Epipolar geometry of the stereo camera assembly in the node. Epipolar geometry refers to the relative separation and orientation of the cameras <NUM>, <NUM> in the stereo camera assembly and is precisely determined by a conventional calibration process not described herein.

After determining the 3D positions of targets within the scene, the processor <NUM> will cause the node to report (i.e. send) the determined 3D positions of the targets to the hub controller <NUM>. The hub controller <NUM> will then determine, for each target, a single global position of the target within the first co-ordinate space <NUM>, based on the 3D positions reported by the nodes for that target.

With reference to <FIG>, the nodes <NUM>, <NUM>, <NUM>, <NUM>, <NUM> are located at different positions about the scene to capture images of the scene from unique viewpoints. Further, the nodes are arranged and oriented such that the fields of view of at least some of the nodes overlap at points within the scene at which one or more targets are expected to be visible to the nodes at some point in time.

It will be appreciated that a target can be any object (or indeed part of an object) of interest that is identifiable in the images using conventional image processing techniques. The target may, for example, be a tool, such as a robot end-effector, which is movable within the scene relative to the nodes. Alternatively, the target may be in the form of a marker or a group of markers that is fixedly attached to a, e.g. moveable, tool such that the position of the tool can be determined based on the position of the marker(s).

In the example of <FIG> and <FIG>, the scene comprises a first target <NUM> which is attached to a moveable robotic arm <NUM> at a fixed position relative to an end-effector <NUM> of the arm <NUM>. A second target <NUM> is attached to a first ground reference point having fixed proximity to the robotic arm <NUM>, a third target <NUM> is attached to a second ground reference point in fixed proximity to a work piece <NUM> or other component to be manipulated or otherwise used in a manufacturing process and a fourth target <NUM> is attached to a third reference point in fixed proximity to the work piece <NUM>.

The positions of the ground reference points and thus the second, third and fourth targets <NUM>, <NUM>, <NUM> in a first co-ordinate space <NUM> are known to the nodes <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and the hub controller <NUM>, such that the position of the movable first target <NUM> in the first co-ordinate space <NUM> can be monitored relative to the positions of the second, third and/or fourth targets <NUM>, <NUM>, <NUM>, based on an analysis of the stereoscopic images of the targets. The positions of the ground reference points are pre-determined by a conventional pre-surveying technique not described herein.

It will be appreciated that by using a system having a network of plural nodes, as compared to a single node or stereo camera assembly, the risk that line of sight to the targets will be lost completely at any point, e.g. during a manufacturing operation, is reduced. Indeed, the end user is able to strategically place the nodes about the scene, to ensure that line of sight to the target(s) is maintained for at least one, and preferably two or more or all of the nodes in the network. This may avoid interruptions in the determination of target positions, thereby enabling continuous monitoring of the targets.

Although five nodes are shown in <FIG> and <FIG>, this is for illustration purposes only. The system may comprise a network comprising any number of two or more nodes, and indeed an advantage of the system is that the number of nodes in the network is readily adjustable to suit a particular scene without requiring modifications to the position determination process. Indeed, an entire production line or even an entire factory can be monitored as a single network.

Additionally, the nodes, in particular their cameras <NUM>, <NUM>, may be configured to detect light in the visible or infrared range of the electromagnetic spectrum. Correspondingly, the flash unit(s) of each node may be configured to illuminate the scene with radiation in a corresponding range of the electromagnetic spectrum.

Using digital cameras and flash units that operate in the infrared range of the spectrum may be particularly advantageous over those operating in the visible range. In that regard, it will be appreciated that ambient visible light is typically a weak and variable light source that restricts the visibility of targets over long ranges. Further, flash units that transmit visible radiation to illuminate targets are often unpleasant or unsafe for people who work within the field of view of the flash units. This is particularly the case if high frequency strobing is used. Infrared radiation, on the other hand, transmits relatively strongly (with greater amplitude) over large distances and does not pose a visible light hazard to people working within range of the flash units. Accordingly, by utilising infrared radiation, it is possible to improve the resolvability of targets within the stereo images whilst avoiding visible light hazards.

Further, while the system has been described above with respect to stereo camera assemblies having only two digital cameras capturing a corresponding pair of stereo images, each stereo camera assembly may have three or more cameras spaced apart by known distances (e.g. to widen the field of view of a respective node). Such a camera assembly would generate a corresponding set of three or more stereoscopic images.

Further still, the system is not dependent on maintaining rigidly mounted nodes - their movement can be computed and compensated without affecting system accuracy. The system can therefore be used in settings where nodes are movable - for example mounted on a moving vehicle or on a drone.

<FIG> schematically illustrates one example of the targets <NUM>, <NUM>, <NUM>, <NUM> that are attached to objects within the scene to be imaged by the system of <FIG> and <FIG>.

The target <NUM> comprises a plate (or generally, a platform) <NUM> having a top surface <NUM> and a bottom surface <NUM> opposite the top surface <NUM>. The bottom surface <NUM> is suitable for attachment to an object within the scene, e.g. by adhesion or other attachment means. On the top surface <NUM> of the plate <NUM>, there is affixed a first group of markers <NUM> which are arranged to define a distinctive pattern, which is recognisable, when imaged, by the processors <NUM>. In this example, the first group of markers <NUM> define a T-shaped pattern, although other patterns are also possible.

The target <NUM> further comprises a moveable marker <NUM> which is suitable for being removably attached to the plate <NUM> at any one of a plurality of attachment points <NUM> located on the top surface <NUM>. In the illustrated example, the moveable marker <NUM> comprises a main body portion <NUM>, a base <NUM> and a pillar <NUM> protruding from a side of the base <NUM> which is opposite the main body portion <NUM>. Each attachment point <NUM> is correspondingly in the form of a hole or indentation which is configured to receive the pillar <NUM> of the moveable marker <NUM>. The cross-sectional profile of the pillar <NUM> and the hole <NUM> are mutually shaped such that the pillar <NUM> may be received snugly and securely within the hole <NUM>, while still being capable of being removed by force.

The attachment points (holes) <NUM> are arranged in an array of predefined positions on the top surface <NUM> of the plate <NUM> such that the position of the moveable marker <NUM> with respect to the first group of markers <NUM> may be selected and varied by the user on a target-by-target basis. Accordingly, each target <NUM> in a scene, such as that described above with respect to <FIG> and <FIG>, may be configured with a unique pattern of markers <NUM>, <NUM>, and therefore allow the processors <NUM> to identify the target <NUM> by its unique pattern. In this way, the target <NUM> can encode data in a visual, machine-readable form by varying the position of the marker <NUM>.

The main body portion <NUM> of each marker <NUM>, <NUM> has a subtantially spherical shape. Such a shape will have a more consistent appearance in images and therefore facilitate quicker and easier detection of the markers over long distances and a wide range of acceptance angles. This is particularly true as compared to circular disk-shaped markers which can appear as ellipses (rather than circles) in the field of view of the node if their camera-facing surfaces are oriented within the scene at an angle relative to the image plane of the cameras. In such cases, automatic recognition of the marker becomes problematic as the ellipse becomes elongated from a more glancing angle. Also the amount of light reflected by a planar disc is severely attenuated when seen at a glancing angle, while spheres avoid this problem.

The individual markers <NUM>, <NUM> may comprise steel and may also or instead be coated with a retroreflective material that is configured to reflect electromagnetic radiation at wavelengths to which the image sensors of the nodes described above are sensitive. Retroreflective materials may be in the form of a paint that comprises glass beads within a substrate.

Although the markers have been described above as being spherical in shape, it is possible for other forms of coded and uncoded markers to be used in the system of <FIG> and <FIG>, and these can be retroreflective, white (or other colours), passive markers and/or active markers, i.e. markers that are self-illuminated. One example of an active marker is an LED light source.

Operation of the system will now be described with respect to <FIG>.

<FIG> is a flow chart schematically illustrating the processing method carried out by each node <NUM>, <NUM>, <NUM>, <NUM>, <NUM> in the system of <FIG> and <FIG>.

The method begins at block <NUM>, at which the node is triggered to capture a pair of stereo images of the scene. The network of nodes may receive a trigger command from the hub controller <NUM> (or other host device), to cause the cameras <NUM>, <NUM> of each node to capture first and second stereo images of the scene simultaneously and in synchronicity with the other nodes of the network.

There may be an initialisation step prior to block <NUM>, wherein the node receives reference data storing information necessary to perform the image processing operation and the corresponding position determination. For example, the node will receive data concerning the targets within the scene (such as the predefined patterns of the markers forming the targets), the positions of the reference points within the first co-ordinate space <NUM> and/or the Epipolar geometry of the stereo camera assembly.

At block <NUM>, the processor <NUM> operates to process the set of stereo images to detect and identify the targets that are visible within the images. The processor <NUM> may use conventional image processing techniques for object recognition to determine which one or more of the targets <NUM>, <NUM>, <NUM>, <NUM> are visible in both the first and second stereo images of the pair. With regard to the specific example of <FIG>, in which each target has a unique pattern of markers, the processors <NUM> can (and if visible will) identify the targets by their unique pattern of markers. If the first target <NUM>, i.e. the moving target of interest, and at least one of the second, third and fourth targets <NUM>, <NUM>, <NUM> located at known reference points within the scene are visible in the first and second stereo images, the processor <NUM> will proceed to block <NUM>.

At block <NUM>, the positions of the targets identified in the first and second stereo images are analysed by the on-board processor <NUM> to determine their 3D positions relative to a reference point on the node in question. It will be appreciated that any conventional photogrammetry technique that is suitable for determining 3D positions of objects in stereo images may be used for this purpose. For example, corresponding targets in the first and second stereo images may be subjected to point triangulation, whereby the processor uses the offset positions of each target (or individual markers) in the first and second stereo images, together with knowledge of the Epipolar geometry of the stereo camera assembly, to determine the 3D positions of the targets relative to the node.

At this stage, the orientation of the targets may be determined based on the relative positions of their individual markers. That is, the node is able to determine not only the position of the targets, but also their pose. In this way, the system is able to resolve the positions of the targets in six degrees of freedom.

The processor <NUM> generates target data that comprises information indicating the determined 3D position of the targets <NUM>, <NUM>, <NUM>, <NUM>. This may include the 3D position of the target as determined based on the 3D positions of the plurality of markers forming the target, or the 3D positions of the markers themselves. The target data may also include the determined orientation of a target. The 3D position(s) is in the form of co-ordinate values, and as such are readily transmittable throughout the system. The target data may also comprise metadata that includes one or more of unique point (or target) identifiers (i.e. names), data quality indicators such as the roundness of measured targets, and the camera settings used for the dataset, all of which can be used to enhance the computation of the global position data, as is described later in reference to <FIG>. The metadata may also comprise time-stamp data associated with the 3D position data, wherein the time-stamp data indicates the time at which the set of stereoscopic images was captured and thus the time at which the target was at the determined position.

At block <NUM>, the target data comprising the position data and metadata for each target identified within the stereoscopic images is sent via the communications line <NUM> to the hub controller <NUM>. The target data may be sent to the hub controller at substantially the same time as the other nodes in the network. If, however, a node was unable to identify targets within the pair of stereoscopic images, the node may be configured so as not to send target data to the hub controller, to save network bandwidth, or otherwise indicate its failure to identify targets to the hub controller.

In embodiments, the nodes are configured to process their respective pairs of stereo images in parallel, such that the overall processing time is reduced. The nodes may operate in parallel by performing any one or more of the functions at blocks <NUM> and <NUM> at the same time as one (or more or all) of the other nodes in the network performs any one or more of the functions at blocks <NUM> and <NUM>.

Although target identification has been described above as being performed as part of block <NUM> before determining the relative position of the targets from the node at block <NUM>, it will be appreciated that the step of identifying a target can be performed after block <NUM>. For example, at block <NUM> the on-board processor <NUM> may simply detect the presence of markers in the images and proceed to determine the relative positions of the markers from the node at block <NUM>, before the markers are then classified and identified as belonging to a specific target before block <NUM>.

Furthermore, although the node has been described above as being configured to report to the hub controller the relative positions of the targets from the nodes, this is not necessarily always the case. For example, in some arrangements, each node may be configured to convert the relative positions (co-ordinate values) of the targets from the node to absolute positions (co-ordinate values) within the first co-ordinate space <NUM> and report the absolute positions to the hub controller in addition to or instead of the relative positions. Thus the target data may comprise information indicating the 3D position of the first target <NUM> and optionally the second, third and fourth targets <NUM>, <NUM>, <NUM> in the first co-ordinate space <NUM>.

As mentioned above, the position of the second, third and fourth targets <NUM>, <NUM>, <NUM> in the first co-ordinate space <NUM> may already be known to the node. In such cases, the absolute position of the first target <NUM> may be determined based on a positional relationship between the first target <NUM> and the first, second and/or third targets <NUM>, <NUM>, <NUM>. The positional relationship may be determined at the processor by comparing the relative position of the first target <NUM> (from the node) to that of the second, third and fourth target(s) <NUM>, <NUM>, <NUM>.

In other embodiments, the node position in the first co-ordinate space <NUM> may be predetermined and known to the node so that the absolute position of the first target <NUM> can be determined based on its relative position from the known position of the node.

<FIG> is a flow chart schematically illustrating the processing steps carried out by the hub controller <NUM>.

At block <NUM>, the hub controller <NUM> receives the target data from the nodes that have identified targets within their pairs of stereo images. It will be appreciated that not all of the nodes will necessarily always transmit target data to the hub controller (e.g. due to obscured line of sight to targets). However, missing or incomplete target data from individual nodes does not prevent the hub controller <NUM> from resolving the 3D positions represented in the target data received from nodes that have identified targets within their pairs of stereo images.

The datastream is substantially continuous (in that it is sent at high, uninterrupted intervals) as long as one node can always see a target. If a target drops out of sight of all nodes, its position will not be determined and sent to the hub controller, but even in that case the target position will be re-determined and sent to the hub controller when it comes back into the line of sight of any one of the nodes.

At block <NUM>, the global position of each target represented in the received target data is determined based on the 3D positions of the target (where the 3D positions for a target may be for the target as a whole or individual markers forming the target) as determined locally at the nodes. The reported positions for a target may differ as a result of inherent inaccuracies of the nodes. Therefore, the target data from the nodes may collectively define a set of 3D points for the target, i.e. a set of plural 3D positions for the first target <NUM>. The hub controller <NUM> is therefore configured to determine a single global position that is representative of the set of 3D points, and in particular a global position that best represents the determined 3D position(s) reported by the nodes. In embodiments, the hub controller performs a best-fit operation on the set of 3D points for each target, to determine the positions of the targets.

Where the target data received at the hub controller from each node comprises the relative 3D positions of the target(s) from the nodes, the hub controller <NUM> will perform an initial step of converting the relative positions from the nodes to absolute positions within the first co-ordinate space <NUM>. This can be done substantially as described above, based on a positional relationship determined between the first target <NUM> and the second, third and/or fourth targets <NUM>, <NUM>, <NUM> located at fixed reference points. The hub controller will proceed to determine a best-fit of the converted 3D positions of at least the first target <NUM> in the first co-ordinate space <NUM>.

Where the target data received at the hub controller from the network of nodes comprises absolute 3D positions of the first target in the first co-ordinate space, the hub controller <NUM> will determine a best-fit of the target positions reported by the nodes.

In both cases, the best-fit operation determines a weighted fit of the set of 3D positions for the target. The weighting used for each 3D position may be selected based on a number of factors, such as the data quality indicators described above. In some arrangements, the weighting used for each 3D position is selected based on the distance between the target and the node which reported the position. The weighting may be inversely proportional to the distance. That is, more weight is given to 3D positions corresponding to nodes that are closer to the target in question. For example, 3D positions that have been derived locally at nodes that are closer to the target are given more weight than 3D positions that have been derived locally at nodes that are further from the same target.

The orientation of the target(s) may also be determined by the hub controller <NUM> based on the target data, e.g. the reported 3D positions of the individual markers, such that the system is able to resolve the positions of the targets in six degrees of freedom. The orientation of a target may be determined based on the relative absolute positions of its markers (which may be arranged in a predefined pattern which is known to the hub controller).

It will be appreciated that by determining a global position of the target based on the 3D positions reported by plural nodes, the accuracy of the global positioning determination is increased as compared to that achieved by conventional systems in which a single node or stereo camera assembly is used.

At block <NUM>, the global positions and/or orientations of the targets in the first co-ordinate space are output. The global positions and /or orientations may be output and used in a variety of ways, depending on the specific application of the system. In the example of <FIG> and <FIG>, the system operates as a robot teaching or control system, wherein the measured global position of at least the first target <NUM> is used to adjust or guide the robotic arm <NUM> and thus the position of the end-effector <NUM>. The position of the end-effector <NUM> may be calculated based on a known positional relationship between it and the target <NUM>.

The processing steps described above with respect to <FIG> may be repeated, e.g. continuously at a set frequency, such that the system can track the global position and/or orientation of the target over time. For example, each node is able to capture and process a pair of stereoscopic images, and the hub controller is able to process the 3D positions reported by the nodes to determine global target positions and/or orientations, at frequencies greater than <NUM>, e.g. up to <NUM>, for example. This is particularly suitable for manufacturing environments where real-time continuous monitoring of moving objects is necessary. Further, by monitoring the global position and/or orientation of the target over time, the system is able to track the velocity and acceleration etc. of the target, e.g. based on time-stamp data associated with the 3D position data,.

Although the technology has been described above with respect to tracking the position of targets by attaching markers to objects within the scene, marker-based tracking is optional. As mentioned above, the targets being tracked may be any objects within the scene that are recognisable in the stereoscopic images.

It will be appreciated that by determining the positions of the targets locally at the nodes and determining centrally at the hub controller a global position and/or orientation of the target based on the determined positions reported by any one or more of the nodes, the system is extensible in that the number of nodes in the network may be adjusted without adversely affecting the functionality and processing speed of the system. Accordingly, the technology described herein provides a photogrammetry system for monitoring the position and/or orientation of targets with not only better accuracy, but also increased flexibility and versatility.

Claim 1:
A method of operating a photogrammetry system (<NUM>) to determine a position of a target (<NUM>),
the photogrammetry system (<NUM>) comprising:
a network of two or more nodes (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) arranged about a scene which includes the target (<NUM>); and
a hub controller (<NUM>) in communication with the network of two or more nodes (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>);
the method comprising:
each of the nodes (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>):
capturing a set of stereoscopic images of the scene using at least two digital cameras (<NUM>, <NUM>) comprised by each of the nodes, in synchronicity with the other node or nodes in the network
processing the set of stereoscopic images to determine a three-dimensional position of the target (<NUM>) and transmitting target data including the three-dimensional position of the target (<NUM>) to the hub controller (<NUM>), if the target is visible within the stereoscopic images; and
the hub controller (<NUM>):
receiving the target data from the network of two or more nodes (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>); and
determining a global position of the target (<NUM>) in the first co-ordinate space (<NUM>) based on the three-dimensional positions included in the target data received from the network of two or more nodes (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>),
characterized in that
determining the global position of the target (<NUM>) comprises performing a best-fit operation on plural three-dimensional positions determined for the target,
wherein the best-fit operation is a weighted fit of the three-dimensional positions, wherein the weighting attributed to each three-dimensional position is based on the relative position of the target (<NUM>) from the node (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) to which the three-dimensional position corresponds.