System, method, and computer program product to provide wireless sensing based on an aggregate magnetic field reading

A system including a plurality of actuation devices with each actuation device configured to have a magnetized part which moves to a designated position representative of a designated manipulation, when at least one actuation device is manipulated, a magnetic sensor array configured to measure a composite magnetic field produced by the magnetized parts of the plurality of actuation devices, an acquisition device configured to acquire magnetic field data from the sensor array indicative of the measured composite magnetic field, a converter configured to convert the acquired composite magnetic field data into analog values, and a processor configured to evaluate to the analog values to determine which of the at least one activation device is manipulated, how the manipulation reflects collective operation of the plurality of actuation devices, and/or to provide a response indicative of the manipulation. A method and a computer program product are also disclosed.

BACKGROUND

Embodiments relate generally to magnetic identification and, more particularly, to a system, method, and computer program product to provide wireless sensing using a magnetic field measurement.

Training and/or simulation systems are used to assist in training users to operate a variety of complex electro-mechanical systems. For example, flight training is a field where training and/or simulation systems are used to assist in training pilots to operate aircraft. Training and/or simulation systems are also being used in a variety of other vast fields, including, but not limited to, operating of vehicular and/or marine systems, medical training, military combat training, etc.

Currently, many training and/or simulation systems are built utilizing equivalent electronic and mechanical components which are a part of the actual system that the training and/or simulation system is built to emulate. Under this approach as actual systems become more complex, so have the associated training and/or simulation systems. Additionally, to further create a more realistic environment of what a user experiences when using an actual system, very high fidelity trainers are being developed to further enhance the training and/or simulation systems. With such systems not only will the user be able to train by operating controls in learning with an immersive environment, but the user will also be able to visually see the effect of her/his decisions made.

By using the same, or similar, electronics as operational systems, current training and/or simulation systems also require similar wiring of components as the actual operational systems. Because of this requirement, the training and/or simulation systems are complex systems which prohibit them from being mobile and flexible enough to be readily disassembled and reassembled in a shortened period, such as within a few hours instead of days for transportation to the point-of-need. Furthermore, because of the current approach of having fully wired training and/or simulation systems, the cost for such systems are increasing as capabilities of actual systems are also increasing.

Thus, users and manufactures of training and/or simulation systems, including immersive human systems, would realize financial and operability benefit from having a technology which reduce expenses and improves an ability to build, assemble, and/or disassemble such systems by utilization of a wireless sensing technology to replace a use of wires or wiring harnesses between user actuated controls and corresponding subsystems.

BRIEF DESCRIPTION

Embodiments relate to a system, method, and computer software code to provide wireless sensing based on detection of a magnetic field generated from an aggregate of devices or components. The system comprises a plurality of actuation devices with each actuation device configured to have a magnetized part which moves to a designated position representative of a designated manipulation, when at least one actuation device is manipulated, and a magnetic sensor array configured to measure a composite magnetic field produced by the magnetized parts of the plurality of actuation devices. The system further comprises an acquisition device configured to acquire magnetic field data from the sensor array indicative of the measured composite magnetic field, and a converter configured to convert the acquired composite magnetic field data into analog values. The system also comprises a processor configured to evaluate to the analog values to determine which of the at least one activation device was manipulated, how the manipulation reflects collective operation of the plurality of actuation devices, and/or to provide a response indicative of the manipulation.

The method comprises providing at least one actuation device of a plurality of actuation devices on a control panel, the at least one actuation device has a front end which is capable of being manipulated on a front side of the control panel which moves a part of the at least one actuation device which is magnetized. The method also comprises actuating the at least one actuation device of the plurality of actuation devices in response to manipulation which causes the magnetized part to move to a designated position indicative of a designated manipulation of the at least one actuation device. The method further comprises measuring a composite magnetic field created by a plurality of magnetized parts of the plurality of actuation devices, acquiring magnetic field data based on the composite magnetic field measured, and converting the magnetic field data to analog values representative of positions of each actuation device of the plurality of actuation devices. The method also comprises determining, with a learning network and based on the analog values, which of the at least one actuation device was manipulated, how the manipulation reflects operation of the control panel, and/or to provide a response indicative of the manipulation.

The computer software code causes the processor to detect, by activating a detection device, a composite magnetic field created by a plurality of actuation devices on a control panel, each actuation device is capable of being manipulated at a front end of the control panel, and is capable to produce a magnetic field, as a part of the composite magnetic field, from a magnetized part of the actuation device which moves to a designated position indicative of a designated manipulation of the at least one actuation device. The computer software code also causes the processor to acquire, by activating an acquisition device, magnetic field data representative of the composite magnetic field created by the plurality of actuation devices. The computer software code also causes the processor to convert the magnetic field data to analog values and determine, with a learning network and based on the analog values, which of the at least one activation device was manipulated, how the manipulation reflects operation of the control panel, and/or to provide a response indicative of the manipulation. The computer software code is on a non-transitory processor readable storage medium.

DETAILED DESCRIPTION

Reference will be made below in detail to embodiments which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals used throughout the drawings refer to the same or like parts. Broadly speaking, a technical effect is to provide wireless sensing (a form of communicating), using magnetic field detection, where magnetic field changes realized from a complete set of instruments or actuation devices are evaluated to determine which of the instruments have been actuated or manipulated. To facilitate an understanding of embodiments, reference to specific implementations thereof is provided herein.

Though embodiments are disclosed herein with reference to the training and/or simulation systems (including immersive human systems), the embodiments are applicable with other devices or systems. As a non-limiting example, the embodiments may be utilized with other systems, such as, but not limited to, arcade or other gaming systems. Thus the descriptions regarding the embodiments provided herein which specifically discuss being used with training and/or simulation systems shall not be considered limiting. Furthermore, though the term “control panel” is used, this term is not intended to be limiting since actuation devices disclosed herein may be housed by any sort of housing device. Thus the term control panel is applicable to any sort of device used to house or hold any actuation device.

Referring now to the drawings, embodiments will be described. Embodiments can be implemented in numerous ways, including as a system (including a computer processing system), a method (including a computerized method), an apparatus, and/or with a non-transitory processor readable storage medium. Several embodiments are discussed below.

FIG. 1shows an embodiment of a system5providing wireless communication (or sensing) based on aggregate magnetic field detection. The system may include a plurality of different actuation devices10. Such actuation devices10may comprise, but are not limited to, a switch12, rotating (or rotary) knob14, lever16and/or push button18. In an embodiment, the plurality of actuation devices10may control one or more functions of a simulated vehicle, truck, aircraft, or device. Each actuation device10, when manipulated, such as but not limited to, by a user, results in motion in a rear, or back side, of a control panel20that corresponds to actuation of the particular actuation device10at a front side of the control panel20(sometimes hereinafter referred to as “front end22”). More specifically, each actuation device10has a first end22or side which may be manipulated and a second end24(sometime hereinafter referred to as “back end24”) or side which moves in accordance with the manipulation of the first end22. The front end22of the actuation device10may be accessible and/or viewable by the user so that the user may manipulate it, whereas the back end24may not be viewable and/or accessible by the user. The type of movement realized may be of any sort, such as, but not limited to, front to back, side to side, circular, etc. A type of manipulation is non-limiting. As several non-limiting examples, manipulation may be caused by a user pulling the lever16, rotating the knob14, pushing the button18, etc., and/or by automatic manipulation based caused by a processor45or computer40(such as a simulation host computer) in response to or as part of a simulated mission or training exercise. Though the computer40and processor45are illustrated as the processor45being a part of the computer40, the processor45may be independent of the computer40.

In an embodiment, the actuation devices10may be faux or dumb actuation devices, requiring no electrical wiring or electrical contact components. The faux or dumb actuation devices10may be moved in a manner similar or identical to real actuation devices. However, the faux or dumb actuation devices10, when actuated, may not provide an electrical signal, stimulus, or make an electrical connection or contact as the real actuation devices would provide. Thus, the use of magnetic fields as taught herein is outside the scope of the electrical connection or contact that the real actuation devices use to interact with and/or which may control one or more functions of an actual vehicle, truck, aircraft, or device.

In an embodiment, the system5may employ actuation devices10which may employ contactless switching. The system5may be configured to sense one, a set or an array of switch states of one, a set or an array of faux or dumb actuation devices.

In communication with the second end24of the actuation device10may be a magnet26, such as, but not limited to, a high gauss magnet. The magnet26may comprise neodymium. A respective magnet26is attached to a moving part of the actuation device10, such as, but not limited, at the second end24. Though the term “magnet” is used herein, this term should be considered as having a part of the actuation device which is magnetized, hence a magnetized part, or producing a magnetic field. Thus the magnetized part is a material that is magnetized and creates its own persistent magnetic field. In another embodiment, the magnetized part may be an electromagnet that acts as a magnet when an electric current passes through a coil of wire. The current may be applied when the actuation device10is manipulated or may be constantly applied. As such, the use of the term “magnet” should not be considered limited to simply a permanent magnet. Furthermore, even though the magnet26is disclosed as being attached to the actuation device10, the magnet26may be an integrated part of the actuation device10which moves when the actuation device26is manipulated. This part may be considered the magnetized part. Therefore, the use or description of being attached shall be construed broad enough to cover the magnet being integrated as part of the actuation device10.

As the moving part of the actuation device10is moved due to manipulation of the actuation device10, such as, but not limited to, by a user, the projection of the magnetic field generated by the magnet26may be changed responsive to the manipulation. As a non-limiting example, if a switch12or lever16is the actuation device10, when the switch12or lever16is moved to a certain position, magnet26may be moved, such as to a specific position, wherein the position is repeated each time the switch12or lever16is placed in the same similar position in subsequent manipulations. More specifically, based on a designated manipulation, the magnet may be moved to a designated position which is indicative of the designated manipulation.

The magnetic fields of all of the magnets26attached to respective actuation devices10combine to create a composite three-dimensional magnetic signature (aggregate magnetic field or composite magnetic field) that represents a state of all actuation devices10, or in general the control panel20. A magnetic sensor array30is provided. The array may comprise a plurality of magnetic sensors. In a non-limiting example, the array30may comprise a two-dimensional array of vector field gaussmeters31. The array30may be placed within view of a back side of the control panel20so that the array30cuts a plane through the magnetic field of the magnets26attached to the actuation devices10. The gaussmeters31are read periodically so that changes in any actuation devices, namely, manipulations, may be detected as scalar variations in one or more individual gaussmeter readings. The gaussmeters31may be read at least as frequently as every ten (10) milliseconds (“ms”), but may be read more frequently (up to being read continuously).

Though disclosed as being located at the back side of the control panel20, the array30may be placed at another location proximate to the actuation devices10and their respective magnets26so that the aggregate magnetic field, is accessible by the array30. Also, though one magnet26is illustrated as being attached a respective actuation device10, more than one magnet26may be attached to each actuation device10. In such an arrangement, a single magnet26may be attached to more than one actuation device10so that when movement or a position change of the one magnet26occurs, the new position may signify which actuation device10has been manipulated. Thus, the disclosure of a one-to-one relationship between the magnet26and activation device10should not be considered limiting.

An amplifier, or amplification device,38, such as, but not limited to, an instrumentation amplifier may be provided to amplify the composite magnetic field signal collected by the gaussmeters31. In another embodiment, the amplifier38may amplify the composite magnetic field prior to the gaussmeters31collecting the magnetic field data. A detector39may also be provided to measure and/or acquire magnetic field data produced by the entire control panel. The amplifier38and detector39may be part of a single component42. The detector39may further convert the sampled magnetic field data into analog values. This may be accomplished every 10 ms or faster. In an embodiment, a converter33may be provided to convert the data into analog values. Though the term “analog values” is used in its plurality, analog values may only include a single analog value. Therefore the use of analog values also comprises a singular analog value.

In an embodiment, the ability to convert the sampled magnetic field data to analog values may be performed by a computer40, more specifically at least one processor45. Thus, the processor45may comprise the converter33part of the detector39or any subsystem or computer program which may convert the magnetic field data to the analog values. The analog values may be evaluated, within the processor45, to determine which one (or ones) of the actuation device10is manipulated based on measurements from the composite magnetic field. The analog values may also be used to determine how the manipulation (more specifically which one (or ones) of the actuation device manipulated) reflects operation of the control panel, and hence the training and/or simulation system. The analog values may also be used to provide a response indicative of the manipulation, such as notifying the user of information (such as, but not limited to, through a display or audible notifier) and/or communicating to the simulation host computer40which may be integral to the training and/or simulation system. As explained above, the processor may be separate from the computer40. Furthermore, the computer40which may operate with the processor45may be distinct and separate from the simulation host computer40. Therefore, the representation of computer40is not meant to be considering limiting with respect to signifying a single computer since a plurality of computers40may exist, each with independent or redundant functions.

The processor45may utilize a probabilistic model to analyze the analog values to determine which one (or ones) actuation device may be, or may have been, manipulated, how the manipulation may reflects operation of the control panel20, and/or to provide a response indicative of the manipulation. A non-limiting example of use of a probabilistic model may be a learning network43, and by way of a non-limiting example, may be a trainable Bayesian network which may analyze the analog values to determine which of the plurality of actuation devices may be, or may have been, manipulated, how the manipulation reflects operation of the control panel, and/or to provide a response indicative of the manipulation. Another non-limiting example is a classifier neural network, namely a neural network which is used to infer a function from observations and able to use it. For simplification purposes only, learning network43may be used to represent the probabilistic model, the Bayesian network, and/or the classifier neural network.

The Bayesian network is a mathematical model that encodes probabilistic relationships among variables of interest. When used in conjunction with statistical techniques, the mathematical model has several advantages for data analysis since it can be used to predict and then learn. For example, once the model encodes dependencies among all variables, more specifically a plurality of magnetic fields taken from a plurality of magnetic field readings where different actuation devices10are manipulated, it may be capable of readily handling situations where some data entries are missing. Thus, the processor45may be able to determine a particular actuation device's setting when the composite magnetic field reading is not determinative based on prior obtained readings, but where the magnetic field readings are based on their respective analog values assigned. This can happen when ambient electro-magnetic noise infiltrates the system.

The Bayesian network may also be used to learn causal relationships, and hence can be used to gain understanding about operation of the training and/or simulation system and predict consequences of manipulation of at least one of the actuation devices. The Bayesian network may comprise both a causal semantic and a probabilistic semantic. Thus, it may be used to combine prior knowledge (former analog values) and current analog values to determine effects of manipulation of any one actuation device of the plurality of actuation devices, which in essence results in a learning network43. Bayesian statistical methods may be used in conjunction with the Bayesian network to provide for an efficient and principled approach for learning how a manipulation affects the training and/or simulation system. Bayesian statistical topologies may also be hybridized with classifier neural network topologies creating a system that more accurately interprets the magnetic data.

FIG. 2shows a block diagram illustrating the system5of an embodiment. As discussed above, the plurality of actuation devices10is provided. The control panel20may house the plurality of actuation devices10. Each of the plurality of actuation devices10may be manipulatable from a front side50(illustrated inFIG. 1) of the control panel20. Magnets26(magnetized parts) may be located on a back side69(illustrated inFIG. 1) of the control panel20. Each respective magnet26may be configured to be in communication with at least one respective actuation device10so that the respective magnet26may move from a first position to at least one other position when the at least one actuation device10is manipulated, the other position being representative of the manipulation. As explained above, the magnet26may be moved to a designated position which is indicative of a designated manipulation of the actuation device10. This movement of the respective magnet26causes a change in the composite magnetic field generated by the control panel20or the plurality of magnets26. The magnetic sensor array30may be within close enough proximity to the control panel20to cut a two-dimensional plane through the composite three-dimensional magnetic field. The amplification device38is also disclosed which may be used to amplify and convert magnetic field data to the analog values.

The computer40is shown having at least one processor45which receives the analog values. The processor45may be used to process the learning network43. Based on the analog values, the computer40and/or the processor45may be connected to the control panel20to communicate, such as to the user, which of the plurality of actuation devices10may have manipulated, how the manipulation reflects operation of the control panel, and/or to provide a response indicative of the manipulation. The computer40may comprise other components that those skilled in the art would recognize as being a part of a computer, such as but not limited to, memory (volatile and non-volatile)51, a bus52, graphics controller53, etc. The learning network43may be located within the computer program instructions47.

To minimize the control panel20affecting the magnetic field of the magnets26, the control panel20may be made of a composite and/or plastic material so as to have a non-magnetic property. More specifically, the control panel may be configured to minimize the magnetic field that may be emitted from the structure of the control panel (excluding the actuation devices). As a non-limiting example, the control panel20may be made utilizing three-dimensional (3D) printing technologies, possibly resulting in the structure being made of a non-metallic alloy. In another embodiment a non-magnetic covering55(illustrated inFIG. 1) may be on, or is a part of the back side69of the control panel20. By completely eliminating or significantly reducing (or minimizing) any magnetic fields attributable solely to the control panel's structure, information obtained or learned from the processor45may be integrated or downloaded into a processor45of other similar or same training and/or simulation systems, thus allowing what has been learned by the processor45from a prior training and/or simulation system to be applicable to other training and/or simulation systems with minimum additional training by the Bayesian network43. Similarly, configuring the structure of the control panel20as explained above shields components internal to the structure of the control panel from external stray magnetic fields created by the magnets26.

FIG. 3shows a flowchart of a method90of an embodiment. The method90comprises providing at least one actuation device of a plurality of actuation devices on a control panel, the at least one actuation device has a front end which is capable of being manipulated on a front side of the control panel which moves a part of the at least one actuation device which is magnetized, at100. The method also comprises actuating the at least one actuation device of the plurality of actuation devices in response to manipulation which causes the magnetized part to move to a designated position indicative of a designated manipulation of the at least one actuation device, at102. The method also comprises measuring a composite magnetic field created by a plurality of magnetized parts of the plurality of actuation devices, at104, and acquiring magnetic field data based on the composite magnetic field measured, at108. The method further comprises converting the magnetic field data to analog values representative of positions of each magnet of the plurality of actuation devices, at110. The method also comprises determining, with a learning network and based on the analog values, which of the plurality of the at least one actuation device was manipulated, how the manipulation reflects operation of the control panel, and/or to provide a response indicative of the manipulation, at112.

The method may further comprise amplifying the composite magnetic field prior to acquiring the magnetic field data to ensure detection, at106. The method may also comprise providing a response to the control panel and/or a user indicative of the manipulation, at114. As discussed above, acquiring the magnetic field data, at108, or converting the magnetic field data, at110, to the analog values may be performed at least as frequently as ten milliseconds. Whereas wirelessly connecting the plurality of actuation devices may comprise communicating between the plurality of actuation devices and the control panel to occur through a processor operating the learning network, at116. Finally, the step of determining, with the learning network, at110may further comprises determining with the probabilistic model, the trainable Bayesian network, or the classifier neural network (collectively referred to as the learning network) which of the at least one actuation device was manipulated, how the manipulation reflects operation of the control panel, and/or to provide a response indicative of the manipulation.

Applying the embodiments disclosed herein would still allow for subsystems that may be tested prior to placing the subsystems in an operation system, such as but not limited to prior to finalizing integration into the operational system. As a non-limiting example, if a new avionics subsystem was developed for an aircraft, the avionics subsystem may be placed in a location reserved for the avionics subsystem on the training and/or simulation system. The avionics subsystem may function as intended within the training and/or simulation system, only the actuation devices would function as disclosed herein where the actuation devices would communicate to the avionics subsystem by first communicating through the aggregate magnetic field system disclosed herein (magnetized moveable material, magnetic sensor array, processor, learning network (as a non-limiting example), etc.).

Persons skilled in the art will recognize that an apparatus, such as a data processing system, including a CPU, memory, I/O, program storage, a connecting bus, and other appropriate components, could be programmed or otherwise designed to facilitate the practice of embodiments of the method. Such a system would include appropriate program means for executing the method. Also, an article of manufacture, such as a pre-recorded disk, computer readable media, or other similar computer program product, for use with a data processing system, could include a storage medium and program means recorded thereon for directing the data processing system to facilitate the practice of the method.

Embodiments may also be described in the general context of computer-executable instructions, such as program modules, being executed by any device, such as, but not limited to, a computer, designed to accept data, perform prescribed mathematical and/or logical operations usually at high speed, where results of such operations may or may not be displayed. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. In an embodiment, the software programs that underlie embodiments can be coded in different programming languages, for use with different devices, or platforms. It will be appreciated, however, that the principles that underlie the embodiments can be implemented with other types of computer software technologies as well.

Moreover, those skilled in the art will appreciate that the embodiments may be practiced with other computer system configurations, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by processing devices located at different locations on board of a vehicle or stationary device, that are linked through at least one communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In view of the above, a non-transitory processor readable storage medium is provided. The storage medium comprises an executable computer program product which further comprises a computer software code that, when executed on a processor, causes the processor to detect, by activating a detection device, a composite magnetic field created by a plurality of actuation devices on a control panel, each actuation device is capable of being manipulated at a front end of the control panel, and is capable to produce a magnetic field, as a part of the composite magnetic field, from a magnetized part of the actuation device which moves to a designated position indicative of a designated manipulation of the at least one actuation device. Also, the computer software code that, when executed on a processor, causes the processor to acquire, by activating an acquisition device, magnetic field data representative of the composite magnetic field created by the plurality of actuation devices, and convert the magnetic field data to analog values. Furthermore, the computer software code that, when executed on a processor, causes the processor to determine, with a learning network and based on the analog values, which of the at least one activation device is (and/or was) manipulated, how the manipulation reflects operation of the control panel, or to provide a response indicative of the manipulation.

Thus, based on the embodiments disclosed herein, a system, method, and computer program product are provided for interrogating controls, identified as actuation devices, on a control panel as a mass operation where the need for panel wiring is reduced or eliminated. The interrogation is accomplished based on a composite magnetic field signature realized from a composite three-dimensional magnetic field created by a plurality of magnets, each respectively connected to a respective interrogating control. Magnetic field data is measured and converted into analog values that are analyzed with a processor employing a process, such as but not limited to, a probabilistic model, a Bayesian network, and/or a classifier neural network (collectively recognized herein as a learning network), to ascertain which control controls) is (and/or has been) manipulated.

While embodiments have been described with reference to various embodiments, it will be understood by those skilled in the art that various changes, omissions and/or additions may be made and equivalents may be substituted for elements thereof without departing from the spirit and scope of the embodiments. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the embodiments without departing from the scope thereof. Therefore, it is intended that the embodiments not be limited to the particular embodiment disclosed as the best mode contemplated, but that all embodiments falling within the scope of the appended claims are considered. Moreover, unless specifically stated, any use of the terms first, second, etc., does not denote any order or importance, but rather the terms first, second, etc., are used to distinguish one element from another. Furthermore, the use of past or present tenses may be used interchangeably and should not be considered as limiting.