The Dark Side of Logic: The near crash of SmartLynx Estonia flight 9001
Tags: Estonia, Aviation, Software Engineering

The Dark Side of Logic: The near crash of SmartLynx Estonia flight 9001 Admiral_Cloudberg Mar 20·21 min read

Firefighters respond to SmartLynx Estonia flight 9001 after its emergency landing. (aviation-accidents.net)

On the 28th of February 2018, two pilots, four trainees, and an inspector departed Tallinn, Estonia for a routine training flight aboard an Airbus A320. But as the last student pilot carried out a touch-and-go landing, the plane was wracked by a series of confusing mechanical failures. The pilots lost all control over the elevators; the plane lost height and scraped the runway on its engines before careening back into the air. The elevators were jammed, the other flight controls were degraded, the flaps weren’t working, the right engine was on fire. Every conceivable warning blared simultaneously. Faced with impending catastrophe, the pilots used what little control they had to line up with the runway for a desperate emergency landing. On final approach both engines failed, but by squeezing out the last dregs of speed, the captain managed to nurse the plane to the very threshold of the runway, where it touched down hard and rolled out in the snow. Through nerves of steel, all aboard had been saved. But what had led to this near-disaster? The answer was a surprise to everyone: there was nothing wrong with the flight controls. The sequence of events actually began when someone used the wrong oil to lubricate an obscure piston deep inside the horizontal stabilizer — a subtle mistake which led to an escalating series of computer failures that nearly cost seven people their lives.

ES-SAN, the aircraft involved in the accident, pictured one week before the event. (Jevgeni Ivanov)

SmartLynx Airlines Estonia is a wholly-owned subsidiary of SmartLynx Airlines, an independent Latvian carrier specializing in charter flights from the Baltic states to holiday destinations. As the Estonian branch of the company, SmartLynx Estonia operates a fleet of three Airbus A320s based out of the country’s main airport in Tallinn, the capital city.

But SmartLynx Estonia flight 9001 was not scheduled to leave the country: in fact, it wouldn’t even get far from Tallinn. The purpose of the flight was to give student pilots working their way through the airline’s in-house A320 training program practice with takeoffs, landings, and go-arounds. During the approximately three-hour session, four trainee pilots with limited flying experience would each take off, perform a go-around, execute five touch-and-go landings, then land for real, under the supervision of an instructor. Also on board were a safety pilot, qualified to fly the A320 in case of an emergency, and an inspector from the Estonian Civil Aviation Authority, who was there to monitor SmartLynx Estonia’s training program. The plane they were flying was 18 years old, but it was new to the airline: SmartLynx had purchased it earlier that month.

Under the watchful eye of the instructor — a 63-year-old veteran pilot with over 24,000 flight hours — the first three trainees carried out their exercises without incident. There was only one small hiccup, an annoying caution message that kept coming back over and over again: “ELAC 1 PITCH FAULT.” The pilots knew this meant there was something wrong with one of the plane’s computers, but the manual said all they needed to do was turn the computer off and back on again, so that’s what they did.

The Airbus A320 is a fly-by-wire aircraft, meaning that pilot control inputs are fed to a bank of computers, which in turn augment those inputs before commanding the control surfaces to move. This allows the plane to fly more smoothly, helps the pilots extract maximum performance and efficiency from the airplane, and prevents them from making inputs that could lead to a loss of control. To ensure redundancy, each set of control surfaces is attached to a different computer, each of which has multiple backups that can kick in if the primary computer fails. Among these computers are the two ELACs, short for Elevator Aileron Computers, which transmit inputs from the pilot and autopilot to the elevators, ailerons, and horizontal stabilizer.

(ESIB)

The ELACs are part of a multi-layered system intended to ensure the integrity of the flight controls at all times. Normally, ELAC 2 is responsible for the elevators, but if it encounters a problem, this responsibility can be transferred to ELAC 1. If both ELACs fail, still not all is lost! At this point a different pair of computers, called the Spoiler Elevator Computers or SECs, which normally control the state of the aircraft’s spoilers, can step in to control the elevators as well. Thus, to lose control over the elevators, four computers must fail at the same time. It’s easy to see why the crew of flight 9001 were not worried when the airplane kept telling them about a problem with ELAC 1.

Unknown to anyone on board, the caution message that they were getting was only the tip of the iceberg. The origin of the problem lay with the Trimmable Horizontal Stabilizer Actuator (THSA). This is the hydraulic actuator which physically moves the horizontal stabilizer, the control surface which the pilots and autopilot use to adjust the angle at which the plane is stable. As opposed to the elevators, which are meant for one-time inputs, the stabilizer trim is for longer-term adjustments that ensure the plane stays at the pitch angle where the pilots or the autopilot want it. Also unlike the elevators, the stabilizer trim is connected mechanically to the pilots’ flight controls, and can still be used even without any functioning computers.

The order in which the four computers take over control of the elevators and stabilizer trim. (ESIB)

Whenever flight 9001 came in for a touch-and-go landing, the wheels would touch the ground, and after five seconds the THSA would enter what is known as the “ground setting.” In order to make the landing rollout easier, upon entering the ground setting the ELAC trims the stabilizer in the nose down direction, assisting the pilot in planting the aircraft on the runway. But in a touch-and-go landing, where the plane will remain on the ground for only a few seconds before taking off again, this automatic adjustment to the stabilizer is undesirable. Thus, in order to assist with the imminent takeoff, the instructor would use the manual trim wheel in the cockpit to override the computer and hold the stabilizer in a nose up position for takeoff. Using the manual trim wheel disconnects the pitch trim actuator (PTA), which sends electrical signals from the computers to the THSA, and in its place connects a device called the override mechanism, which inserts itself downstream of the PTA and transfers pilot inputs to the THSA instead of computer inputs.

Meanwhile, the ELAC constantly compares the commanded stabilizer position with its actual position in order to detect any mechanical failure in the system. It accomplishes this by matching values from two data channels: the command channel, which transmits the commanded position, and the monitoring channel, which transmits the actual position. If a discrepancy between the two lasts more than one second, a fault is triggered. In its default state, this very simple function would be unable to tell the difference between a pilot overriding the autopilot using the trim wheel and failure of the horizontal stabilizer, because the computer will continue to send signals to the PTA despite the fact that it is no longer connected to the stabilizer. To rectify this, whenever the pilot applies torque to the trim wheel in the cockpit, an override piston inside the PTA moves downwards and contacts three microswitches, which transmit a signal telling the computers to stop comparing the command channel with the monitoring channel.

(ESIB)

However, this system on this particular A320 contained a tiny flaw: the oil used on the override piston was twice as viscous as the oil called for in the manual. As a result, the friction on the override piston was too high, and it would sometimes fail to extend far enough to contact the microswitches. This meant that during many of the touch-and-go landings, the instructor would override the computers by using the manual pitch trim wheel, but the microswitches wouldn’t make contact with the piston, and the computer would continue comparing the command and monitoring channels. Detecting that its commands were not affecting the stabilizer position, ELAC 1, which normally controls the stabilizer, would register a fault and shut down. Its duties would be transferred to ELAC 2, and an “ELAC 1 PITCH FAULT” caution message would appear on the screen between the two pilots.

(ESIB)

Because the failure of one ELAC is considered an advisory matter without major safety implications, this message would only appear after the plane climbed through 1,500 feet after each touch-and-go landing, so as not to distract the crew during takeoff. Every time this message appeared, the pilots simply turned ELAC 1 off and back on again and the problem would go away. But on one of the touch-and-go landings with the third trainee, the pilots failed to notice the caution message and never reset ELAC 1.

As the fourth trainee came in for their third touch-and-go some minutes later, ELAC 1 was still off, and the stabilizer trim was being controlled by ELAC 2. As the instructor grasped the trim wheel to prevent the stabilizer from moving nose down, the same sequence of events happened again: the microswitches didn’t connect, the computer detected a discrepancy between the commanded position and the actual position, and ELAC 2 registered a fault and shut down.

With both ELACs inoperative, control over the elevators and the horizontal stabilizer was transferred to SEC 1 (Spoiler Elevator Computer 1). But a remarkable coincidence was about to plunge everyone on board into much greater danger.

The main role of the SECs is to facilitate the automatic deployment of the spoilers upon touchdown. The spoilers are panels that lift up from the wings to reduce lift and force the plane into the runway. In order to determine whether the spoilers should be deployed, the SECs constantly monitor sensors in the landing gear that detect whether the plane is on the ground or in the air. Although the pilots did not arm the spoilers for the touch-and-go landings, the SECs all the same continued to monitor these sensors.

The SECs get their landing gear information from a pair of computers called the Landing Gear Control Interface Units (LGCIUs). Like the ELACs, the SECs have a command channel and a monitoring channel in order to detect failures in either LGCIU. The command channel is supplied from LGCIU 1, while the monitoring channel receives data from LGCIU 2; thus, if one unit says the plane is on the ground while the other says it’s in the air, the SECs will detect this discrepancy and shut off, preventing the spoilers from automatically deploying in flight due to a faulty indication by one of the LGCIUs.

(ESIB)

At almost the exact moment that control over the elevators and horizontal stabilizer was transferred to the SECs, the plane bounced off the runway and became airborne for approximately one second. By sheer coincidence, this event poked a hole in the logic used by the SECs to determine whether there was a fault with the LGCIUs. Although both LGCIUs send data about the plane’s air/ground status to the SECs every 120 milliseconds, they are not synchronized, meaning they don’t send this data at the same time.

If the SECs receive indications from both LGCIUs that the plane is in the air for a period of one second or more, the SECs switch to “flight mode.” Flight mode remains active for a minimum of 20 seconds. But because the command and monitoring channels do not sample data synchronously, it was possible for one channel to detect that the plane was airborne while the other channel did not, as long as the plane was in the air for only slightly more than one second — equivalent to about nine 120-millisecond sampling intervals, after rounding. Therefore, if the plane became airborne for, say, 1.15 seconds, it was possible for either nine or eight complete sampling intervals to fall within that 1.15-second period, depending on the exact timing of the intervals (for a more detailed explanation of this example, refer to the diagram below). So if the intervals line up such that the command channel detects an airborne status over nine intervals and the monitoring channel detects an airborne status over eight intervals, the command channel will see that the one-second threshold has been met and will switch to flight mode for 20 seconds, while the monitoring channel detects an airborne period less than one second and stays in ground mode. With one channel in flight mode and the other in ground mode, the SECs believe that there has been a failure of one of the LGCIUs, and they both shut off (since they both get their data from the same source).