Patent Publication Number: US-2021192969-A1

Title: Flight Simulator with a Visual System Integrated in a Robotic Manipulator

Description:
CROSS-REFERENCES TO RELATED APPLICATIONS 
     This application claims priority from U.S. provisional patent application No. 62/949,919, filed Dec. 18, 2019 entitled “Flight Simulator with a Visual System Integrated in a Robotic Manipulator”, incorporated by reference for all purposes. 
    
    
     STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
     N/A 
     FIELD 
     The technology herein relates to flight simulators, and more particularly to flight simulators within a visual system integrated in a robotic manipulator. 
     BACKGROUND &amp; SUMMARY 
     A flight simulator artificially simulates aircraft flight and the environment in which the aircraft flies. Flight simulators are often used as a critical step in testing new aircraft designs. They often replicate the control laws that govern how aircraft fly, how the aircraft reacts to applications of flight controls, the effects of other aircraft systems, and how the aircraft reacts to external factors such as air density, turbulence, wind shear, cloud, precipitation, etc. Flight simulation is used for a variety of reasons, including flight training (mainly of pilots), the design and development of the aircraft itself, and research into aircraft characteristics and control handling qualities. See “FAR 121 Subpart N—Training Program”, 14 CFR 121. 
     While some training type flight simulators are primary electronic, many flight simulators used to test aircraft design include physical platforms for changing the position and orientation of the aircraft assembly. One prior flight simulator approach is to use the so-called Stewart Platform in a parallel hexapod configuration with 6 linear hydraulic or electrical actuators. The Stewart platform design is extensively used in flight simulators, particularly in the full flight simulator which requires all 6 degrees of freedom (“DOF”). In Stewart Platform based flight simulators developed by Redifon, the payload is often a replica cockpit and a visual display system, normally of several channels, show the outside-world visual scene to the aircraft crew that are being trained. The replica cockpit is mounted on the Stewart Platform, which positions and moves the replica cockpit in 6 DOF. See e.g., Maria Rosa Ruiz, Design and Analysis of a Stewart-Platform-Based Six-Axis Load Cell (MIT June 2017); CN104731103B; CN203084972U; DE102010035814; WO2020117037; U.S. Pat. Nos. 4,016,658; and 4,390,253. While such efforts have been made in the past, they often have been unduly expensive and have not resulted in a high-fidelity representation of actual flight. 
     It would be highly desirable to develop a robotic based flight simulator with fidelity level representative of a real aircraft, with an optimized aircraft product development cycle. Such a system would be able to evaluate the criticality during a flight campaign, optimize product development and provide training in a Flight Simulator having high fidelity representation and low operation cost, providing advantages including:
         Multiplatform Flight Simulator having high fidelity and low operation cost (i.e., modular construction so the same flight simulator can be used interchangeably with multiple different cockpits)   Optimize the product development cycle   Optimize the development Flight Mechanics and control law (CLAW) performance   Application for Upset Recovery condition.       

     Once depicted through the figures and descriptions above it is important to emphasize that the present non-limiting technology has the following advantages among others: 
     High fidelity flight mechanic model in a less complex assembling flight simulator—when compared with a hexapod level D configuration described above. The high-fidelity model is implemented in this type of flight simulator with much less complexity assembling than a Stewart platform. 
     Motion-based system in a less complex flight simulator when compared with a hexapod flight simulator configuration. As one implementation uses a robot arm, the configuration is much simpler than a Stewart platform, which has 6 hydraulic actuators, and limited degree of freedom when compared with a robotic arm degree of freedom. 
     Contribution to a reduced development cycle product. Nowadays a flight simulator is an essential tool to support the aeronautic development cycle. A low and/or middle cost flight simulator, as this in a robotic arm, having a representative visual system, will reduce the time of technical development. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows main constituent systems of example embodiments. 
         FIG. 2  is a Global View of an example non-limiting Visual System at the Robotic Manipulator. 
         FIG. 3  shows an example non-limiting SIVOR visual projection system. 
         FIG. 4A  shows an example mechanical arrangement of projectors and spherical cap screen of SIVOR&#39;s visual system. 
         FIG. 4B  shows, from inside an example cockpit, the external environment projected on the projection screen. 
         FIGS. 4C-4H  show additional details of the cockpit and the projection screen. 
         FIG. 5  shows example mechanical cases to support projectors in an upside-down position. 
         FIG. 6  shows example mechanical integration of the visual system. 
         FIG. 7  shows an example software layer architecture of the visual projection system. 
         FIG. 8  shows an example hardware level architecture of the visual projection system. 
         FIG. 9  shows an example visual blending-warp adjustment along with visual system integration. 
         FIG. 10  shows an example Cooperative Robotics Flight Simulators. 
     
    
    
     DETAILED DESCRIPTION OF EXAMPLE NON-LIMITING EMBODIMENTS 
     One example non-limiting example provides a full representative part 25 commercial aircraft flight deck with a visual system integrated with a robotic manipulator to provide an immersive simulation environment for training and research purposes. Such technology provides an architecture that provides visual blending-warp adjustment along with visual system integration, including for example:
         Spherical screen design   Structural design   Projectors allocation (i.e., allocation of plural projectors to different parts of the spherical screen).       

     Based upon this mindset, a visual system complies with Level 7 and D requirements of the FAA&#39;s 14 CFR Part 60. With this kind of integration, requirements such as visual latency, spherical structures, and field of view in horizontal and vertical direction are provided in order to operate with the robotic motion platform and resulting in flight simulator fidelity capable of complying with a high-fidelity flight simulator. 
     Another design point is the structural design and payload optimization. The total weight and global structure are designed without jeopardizing the dynamic behavior of this type of flight simulator. 
     The flight simulator based on robotic manipulator exists in one embodiment for researching purpose only without visual system integrated on the cockpit frame which complies with Level 7 and D requirement of FAA&#39;s 14 CFR Part 60. The reason is the payload constraints which gives, in a first glance, some challenging issues to qualify this type of flight simulator. Along these lines, the example non-limiting technology herein provides a structural design strategy that allows the integration of a commercial jet full representative flight deck with a visual system in a robotic manipulator. The mechanical structure allows the integration of the embedded visual system, cockpit mockup and systems&#39; components by means of the techniques DTW (“Design to Weight”, DOI: https://doi.org/10.1007/s40430-013-0109-8) and DTCG (“Design to Center of Gravity”, DOI: https://doi.org/10.1007/s40430-018-1183-8). This strategy overcomes the payload restrictions imposed by the robotic motion platform and the development is done for a robot of the shell that solves manufacturing aeronautic problems. 
     Using the DTW and the DTCG techniques, several iterations of structural Finite Element Analysis and Robot Payload Analysis may be carried out for each new proposition of geometry, dimensions and materials. A topological analysis, which considers different distribution of mass, materials and geometry of the screen according to the main required resonant modes and geometric projection requirements, may be carried out to evaluate the best proposition from the structural point of view. 
     Upon these analyses, the structure may be configured as single linked rigid body, comprised of a carbon fiber screen and carbon fiber trusses. 
     An example embodiment provides an adjustable-tensioning mechanism in each interaction joint in each pair of tubes/screen sections, which is based on a tensional bolt-nut mechanical arrangement that allows stretching the assembly to adequate or enable its vibrating/compliant modes. 
     The final design may be validated in terms of total mass and equivalent center of gravity position by means of a dynamic model of the robot arm that shows that each joint actuator is not subject to an excessive torque for the maximum robot acceleration condition. 
     Example Non-Limiting Overall System 
     With these characteristics as background, the main constituent systems of an example embodiment 5 are pictorially described in  FIG. 1 . Features of each system  5  are described below. 
     Robotic platform  10  consists in one embodiment of a six degree of freedom anthropomorphic robotic arm (e.g., KUKA KR 1000), with 1 ton payload capacity and a 10 meters linear unit (rail)  100  the robot can move upon. The robotic system  10  therefore has seven degrees of freedom (pitch, yaw, roll, up/down, forward/backward, left/right, and linear movement along the rail). The seventh degree of freedom introduced by the linear rail  100  adds complementary features that increase simulator fidelity since the linear rail  100  is responsible to enhance the forces sensed on the pilot body due to forward and backward displacement. Example movement ranges of the different degrees of freedom axes may be: 
     
       
         
           
               
               
               
               
               
               
               
               
             
               
                   
               
               
                 Axis 
                 1 
                 2 
                 3 
                 4 
                 5 
                 6 
                 7 
               
               
                   
               
             
            
               
                 Range 
                 ±150° 
                 +15/−130° 
                 +145/−110° 
                 ±350° 
                 ±118° 
                 ±350° 
                 7 m 
               
               
                   
               
            
           
         
       
     
     Cockpit  20  (which is mounted to and suspended by the robotic platform  10 ) consists of an adaptive environment composed by simulated components and an inceptor with control loads that can reproduce an actual aircraft such as for example Embraer&#39;s Legacy 500 aircraft. The cockpit  20  systems integration extrapolates the simple component juxtaposition by providing their physical and logical integration. In addition, some supplementary functionality may also be implemented. For example, such supplementary functionality may include force feedback in pedals, side sticks and levers, in which the pilot receives feedback from the reaction of the control surfaces of the simulated airplane in a conventional manner. See  FIG. 4C . The system further includes a processor and associated memory storing instructions that when executed by the processor, cause the processor to control the robotic platform  10  in response to the control inputs from the cockpit. For example, when a pilot operates an inceptor in the cockpit  20 , the robotic platform  10  may respond by changing the orientation of the cockpit (e.g., pitch up, pitch down, roll, yaw, etc.) 6 DOF and path control (i.e., Robot Control, PLC Control, Motion Control e.g. KUKA.CNC and Safety Control) of the robotic platform  10  may be accomplished using conventional programming and automation components such as described in “Kuka the control system of the future KR C4” (2016 Kuka.com) and U.S. Pat. No. 10,780,587, each incorporated herein by reference. An example robotic controller for example enables direct programming and operation of robots via G-code, can process complex programs from CAD/CAM and other systems, and can provide high accuracy due to CNC path planning. 
     Visual system  30 : In order to comply with the Level 7 and D requirement of FAA&#39;s 14 CFR Part 60 requirements for a projection screen  30 ′ of a full flight simulator, the visual system  30  in one embodiment has, at least, 1800 in a horizontal and 400 in a vertical direction field of view. In one example embodiment, the virtual system  30  provides a panoramic semi-circular (180 degree field of view) spherical viewing surface in the direction of the horizon, with 20 degrees both up and down from the horizon. The visual system  30  designed for example embodiments has in one example a spherical cap of 4.0 meters of diameter that spans a field of view of 188° horizontal and 48° vertical. It is further separated into 4 different sections which are joined through screws and composite resin. Each screen section is directly linked to the main composite base structure through an array of carbon fiber tubes  170 . See  FIGS. 3, 4D, 4E, 4F, 4H . The tubes  170  provide a lightweight support structure attached to robotic arm  10  that supports the screen  30 ′ in a spherical wrap-around configuration and maintains constant distance between the screen and the viewpoints of the pilots within the cockpit  20  irrespective of movement and repositioning of the robotic platform  10 . In one embodiment, the visual system  30  support structure is mounted to the robotic arm  10  but not to the cockpit  20  so that the same visual system can be used interchangeably with multiple different types of cockpits for simulating different aircraft. See  FIG. 10 . As  FIG. 4B  shows, the visual system  30  simulates a panoramic view of the environment outside the cockpit  20  so that pilots looking out the windows of the cockpit see a realistic simulated view of the world outside the cockpit (e.g., runways, terrain, clouds, other aircraft, fog, etc.) 
     In one non-limiting embodiment, the visual system  30  comprises a set of projectors  32   a ,  32   b ,  32   c  and a projection screen  30 ′, designed to have provision for complying Level 7 and D requirement of FAA&#39;s 14 CFR Part 60 flight simulator, and both embedded in the integrated robot and cockpit system, optimally designed in terms of weight and gravity of center positioning. 
     In one embodiment shown in  FIGS. 2, 3, 4A, 4F and 5 , three connected projectors  32   a ,  32   b ,  32   c  guarantee the uniformity of the generated external images. The visual system  30  in one embodiment has the capability of merging images and unwarping warped images to compensate for visual distortion caused by the projectors and/or the screen  30 ′. See  FIG. 9 . The size of the projection screen  30 ′, especially the vertical direction, is linked to the minimum projection lens distance in one embodiment to ensure sufficient light intensity and reduce defocusing. 
     Supervisory system  40  comprises the physical and logical integration between the robot  10 , the cockpit  20  and visual systems  30 . The supervisory system  40  provides communication in real time and safeguards to guarantee simulator integrity. Considering only two of these systems and its inherent complexity, namely: the aircraft model and the supervisory system, many concerns regarding reliability and safety are addressed. Additionally, the highly coupled architecture and the impossibility to model the entire system of systems would be a favorable environment for detrimental emergent behaviors. This scenario has paved the way for a design approach called Quick Responsive Development and continued evolving strategy. 
       FIG. 2  shows an example embodiment presented as a robotic manipulator  10  with a cockpit  20 , scale one to one, of an aircraft such as an Embraer business jet, and the visual system  30 . In this figure is presented each element that composes the robotic flight simulator. A rail  100  is designed to give a high fidelity acceleration at the pilot body. Item  10  is an off the shelf robotic arm, here a KUKA KR 1000, having payload of 1 ton. Item  30  is the flight simulator screen, which is integrated at the robotic arm  10 . Item  20  is the real cockpit of for example an Embraer business jet. Items  32  is a set of three projectors  32   a ,  32   b ,  32   c  responsible to project the image on the spherical screen  30 . Other embodiments can use different numbers of projectors  32 , or other display technology such as making the screen  30 ′ out of OLED technology. In this example, the screen is curved and configured as a semicircle or a portion of a cylinder to provide a panoramic (“wraparound”) view. 
     The visual system  30  of the Flight Simulator shown in  FIG. 3  is designed to be a lightweight solution compactly coupled around the simulator&#39;s cockpit  20  to be fully coupled in a robotic arm  10  to provide an embedded out-of-the-flight deck view, being able to be jointly moved with the remaining portions of the system along with the flight simulation. Three projectors  32  (e.g., Barco F50 WUXGA), each one using an ultrashort lens (e.g., Barco EN57), are tied to the simulator&#39;s mechanical frame to provide a short distance projection lower than 1939 mm. See Barco, “F50 WQXGA Compact 120 Hz, single-chip DLP projector with WQXGA resolution” (17 Jun. 2019). 
     The projectors  32  are in one embodiment mechanically arranged with a spherical cap projection screen  30  that spans a continuous visual field-of-view of 188° horizontally and 400 vertically, centered on the zero degree azimuth line relative to the aircraft fuselage on the mid-symmetry plane of the cap, being therefore fully compliant with Level 7 requirement of FAA&#39;s 14 CFR Part 60. An example mechanical arrangement is depicted in  3 ,  FIG. 4A, 4D, 4E, 4F, 5 . 
       FIG. 4B  shows, from inside the cockpit  20 , the external environment projected on the projection screen  30 . To suit the standard light beam projection direction of the commercial-of-the-shelf projectors  32  to the required mechanical arrangement of the visual projection, special lightweight mechanical support cases, fully manufactured in carbon fiber, are designed to support the projectors in the upside-down position, as depicted in  FIG. 5 . 
     To comply with lightweight requirements, the visual system  30  mechanical structure may be fully manufactured from composite material, with the screen divided into six 31.3° horizontally pieces that are appropriately mounted along with the mechanical integration, as depicted in  FIGS. 4D, 4G and 6 . 
     From the system&#39;s perspective, in the software level, the aerodynamic model, which runs from an independent engine-software apart in a special computer  200 , feeds a visual server computer  202 , which runs three parallel instances of a commercial-off-the-shelf visual engine software (X-Plane  10 ), as depicted in  FIG. 7 . The visual server  202  for example maintains a common 3D model of a virtual world, and responds to inputs from inceptors and other controls within cockpit  20  as well as to commands or sensors indicating the current position and orientation in 7 DOF of the robotic arm  10 . 
     Each of these visual engine software instances feeds an independent commercial-off-the-shelf image microprocessor (MIPS)  204  that renders the visual projection for a specific projector  32 . The microprocessors  204  include memories storing instructions that when executed by the microprocessors, control the microprocessors to generate and correct images for display by the projectors  32 . Each microprocessor  204  (which may also comprise a conventional graphics processing unit including a 3D graphics rendering pipeline) produces a portion of a panoramic image and sends that panoramic image portion to the projectors  32  for display. 
     On the hardware level, the visual server computer  302  feeds the three MIPS  204   a ,  204   b ,  204   c  in parallel through a multichannel DVI or other interface, which then devotedly render the projection for each projector  32   a ,  32   b ,  32   c  and sends the appropriate image signal to them through an optical-fiber pair interface, as depicted in  FIG. 8 . 
     Each MIPS  204  runs a dedicated programmable firmware that performs blending-warp routines to ensure synchronization, smoothness overlap and the continuity of the final projection among the three projectors  32   a ,  32   b ,  32   c  over the spherical cap field-of-view of 188° horizontally and 40° vertically. Such techniques provide a seamless panoramic image for display on screen  30 ′. The adjustment of the blending-warp, using the aforementioned software and hardware setup, onto the projection mechanical setup is carried out as a final integration step, as depicted in  FIG. 9  showing a test grid displayed on screen  30 ′ that can be used for calibrating the blending warp algorithms. 
       FIG. 10  shows an alternative embodiment which comprises a modular system including two robots  302 ,  304 . Due to the simplicity flight simulator concepts, the flight simulator may be used in a potential flight simulator training center, in such a way different types of cockpits  20  and equivalent configurations software might be set up in very efficient and fast manner. For example, different cockpits  20  modelling different aircraft can be interchangeably swapped in and out for use with a common display system  30  and robotic system  10 . In the example shown, a frame supporting display screen  30 ′ can be modularly interchangeably attached to a frame supporting a cockpit  30 . Alternatively, each different integrated combination of cockpit  20  and display system  30  may be used with a common robotic platform  10 . 
     Alternatively, the flight simulator may use cooperative robotic system in order to allow one robot dedicated to the motion based functions (comprising the cockpit) and the other robot dedicated to the screen based functions. For example,  FIG. 10  shows one robotic platform  304  that supports and positions the visual system  30  and another robotic platform  302  that supports and positions the cockpit  20 . Such alternative using cooperative robotic system may be applied when payload limitation becomes critical or important. 
     Additional Potential Application: 
     Upset Recovery application. Example implementations of this type of flight simulator have more degrees of freedom than a traditional Stewart Platform based flight simulator, and—when integrated with a high-fidelity visual system—the potential to evaluate the pilot reaction and support the validation of modern techniques of aerodynamic coefficients in this flight condition is extremely high. 
     Furthermore, the technology herein provides a disruptive visual system  30  which complies with Level 7 requirement of FAA&#39;s 14 CFR Part 60 requirements. This technology integrates an optimal structure that allows assembly—in a flight simulator—of a screen complying with the Level 7 requirement of FAA&#39;s 14 CFR Part 60 requirements. 
     All patents and publications cited above are incorporated by reference. 
     While the invention has been described in connection with what is presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not to be limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.