Run More Trick Plays. An analysis of NFL trick plays... | by Blake Atkinson | Towards Data Science

Trick plays in football are one of the most fun parts of the game. Double reverses, fake field goals, and flea flickers usually make highlight reels. They are high variance in nature: either you catch the defense off guard and score an easy touchdown, or you have your running back throw a duck into double coverage and give the other team the ball on a silver platter. The high-variance nature of trick plays is intuitive to anyone who watches a lot of football. What might not be intuitive is their actual value. Until recently, the data required to objectively evaluate trick plays was not easily accessible to the public. Thanks to great work by Ron Yurko, and great tutorials by Ben Baldwin and Github user @Deryck97, any schmuck such as myself can spend a couple hours analyzing the value goofy plays over a 10 year period.
How do you capture the value of a play? Obviously a 50-yard completion is worth more than a 2 yard run in almost all cases. However, is a 4 yard run on first down worth more than a 1 yard 3rd down conversion? That’s a difficult question to precisely answer. It’s clear however that not all yards are created equal. A screen pass to end the half against prevent defense might rack up 30 yards as time runs out. All that does is pad stats. Any NFL coach would rather have a 3rd down conversion in a key situation no matter how short. Ron Yurko authored a paper detailing an Expected Points Added metric. In the paper (found here), he is able to determine the amount of points a team is expected to score from any point during the game. If, after a play, that expected points metric is higher, then the play added expected points. Of course, a play such as a sack or interception will result in negative EPA.
Obviously, 90+ yard offensive plays will have the highest EPA and Redzone turnovers have the worst EPA. Here are the top 5 EPA plays in the dataset:
Description (3:15) (Shotgun) B.Roethlisberger pass deep middle to J.Smith-Schuster for 97 yards, TOUCHDOWN.  EPA 9.5Description (10:06) (Shotgun) C.Johnson up the middle for 91 yards, TOUCHDOWN.  EPA 9.3Description (7:44) (Shotgun) C.Palmer pass short middle to M.Floyd for 91 yards, TOUCHDOWN.  EPA 9.2Description (8:51) (Shotgun) P.Mahomes pass deep middle to D.Robinson for 89 yards, TOUCHDOWN. PENALTY on KC-D.Robinson, Taunting, 15 yards, enforced between downs.  EPA 9.1Description (11:51) (Shotgun) D.Carr pass deep middle to A.Cooper for 87 yards, TOUCHDOWN.  EPA 8.9
You get the idea. Here are the bottom 5:
Description (12:14) (No Huddle, Shotgun) M.Vick to WAS 10 for -6 yards (R.Kerrigan). FUMBLES (R.Kerrigan), touched at WAS 10, RECOVERED by WAS-D.Hall at WAS 25. D.Hall for 75 yards, TOUCHDOWN. Lateral batted by 91 - Kerrigan The Replay Assistant challenged the backward pass ruling, and the play was Upheld.  EPA -12.8Description (11:14) (Shotgun) J.Tuel pass short right intended for T.Graham INTERCEPTED by S.Smith at KC 0. S.Smith for 100 yards, TOUCHDOWN.  EPA -12.7Description (6:05) (Shotgun) M.Cassel sacked at DEN 25 for -17 yards (M.Haggan). FUMBLES (M.Haggan), RECOVERED by DEN-J.Hunter at DEN 25. J.Hunter for 75 yards, TOUCHDOWN.  EPA -12.1Description (2:51) M.Vick pass short left intended for B.Celek INTERCEPTED by P.Robinson at NO 1. P.Robinson for 99 yards, TOUCHDOWN. PENALTY on PHI-M.Vick, Low Block, 15 yards, enforced between downs. The penalty for a low block during the interception return will be assessed on the ensuing kickoff.  EPA -12.1Description (7:50) (Shotgun) E.Manning pass short left intended for H.Nicks INTERCEPTED by B.McCann at DAL -1. B.McCann for 101 yards, TOUCHDOWN.  EPA -12.1
EPA is a funny thing, since on offense it can be greater than a touchdown. That makes sense, because if you’re backed up against your own endzone, you’re expected 1) not to score on the drive and 2) the other team is more likely to score on their next drive because of the short field. It can also be -12, because in the redzone 1) you’re expected to score around 5 points on average, and 2) the next possession for the defensive team is supposed to be post-kickoff, probably on their own half of the field.
Now that we have a metric for the value of a play, we can start grouping types of plays together and comparing them. For example, taken from @Deryck97’s tutorial linked above, a comparison of rush vs. pass plays in 2018:
So, using nflscrapR data, how do we know if a play is a trick play? Unfortunately there’s no easy indicator variable that says ‘trick play’. Instead, we have to come up with creative ways to identify them.
The easiest plays to start with are fake kicks. Most play descriptions contain the formation of the play. If a play is labeled ‘Punt formation’ or ‘Field Goal formation’, yet the play type is a run or a pass, then it is likely to be a fake. I had to manually filter some other play types, like when the punter muffs a snap and aborts the play, but that was pretty easy. Here’s the python code:
punts = df.loc[df.desc.str.contains('punt|Punt')]fake_punts = punts.loc[(punts['play_type']=='run')|(punts['play_type']=='pass')]# drop enforced penaltiesfake_punts = fake_punts.loc[fake_punts.penalty!=1]# drop non enforced penaltiesfake_punts =fake_punts.loc[~fake_punts.desc.str.contains('Penalty')]# drop punter mishandles fake_punts=fake_punts.loc[~fake_punts.desc.str.contains('Aborted|Shotgun')] # drop plays where punter intentionally runs out of end zone (not likely to fake a punt in own redzone)fake_punts = fake_punts.loc[~fake_punts.desc.str.contains('SAFETY')] # drop plays without descriptionfake_punts.dropna(subset=['desc'], inplace=True)
I then apply the same filters to plays that did end in an actual punt. There ended up being 177 instances of fakes that I could pick up out of about 22,000 punt attempts (regular season and playoffs). A fake rate of less than 1%!
Let’s come up with some baselines as well. I’ll show the average EPA of a punt, the average EPA of a pass play, the average EPA of a run play, and the average EPA of a pass by Tom Brady 🐐.
Wow! Fake punts are worth about seven times as much as a Tom Brady pass. Of course, the standard deviation of the EPA is much higher too. The standard deviation of the EPA of non-fake punts were 1.16, while the standard deviation of fake punts was 3.17. Obviously, if you fail a fake, then you have a large negative EPA.
Are fake field goals just as valuable? According to my simple analysis, no. Throwing out fumbles, bad snaps, penalties, etc — the EPA of a fake field goal is 0.06. That’s higher than your average pass, but not super exciting since I ran it after the fake punt numbers. I believe this is because what you’re giving up by forgoing a field goal. If you fail a fake field goal, you lose a few expected points you would’ve gotten from the attempt. If all you’re doing is getting a first down, you’re not even gaining that many expected points. They have a similarly large standard deviation like the fake punts. So if you’re trailing a few points, you still might be better off rolling the dice and going with a fake.
Trick run plays and pass plays are not as easy to isolate unfortunately. As a disclaimer, the point this article was to start playing with the NFL play-by-play data. I’m not writing a dissertation on fake plays. So while I did go to lengths to isolate trick plays successfully, I’m not claiming 100% accuracy or the best possible process.
Let’s start with passes. I began by throwing out obvious plays (penalties, two-point conversion attempts) as well as plays I’ve already analyzed (kick formations). I also threw out plays with extreme win probabilities (<5% & >95% win probability). The idea there is to avoid desperate late-game lateral plays like the hook and latter.
I then searched play descriptions for flea flickers and passes by players with less than 30 attempts in the data set. Flea flickers cover laterals to the QB, and in theory, any other trick pass would involve a lateral to a running back or wide receiver that doesn’t normally pass it. Finally, I black-listed QBs that only came in for a very brief period of time and were picked up by my filters. They were easy to find because a whole series of plays sticks out in the descriptions.
Almost there. For run plays, I isolated plays that had a description of lateral or reverse. Luckily, there weren’t many cases of defensive players lateraling after a fumble recovery. I ended up with a sample of 180 plays (manually removed a weird one or two). I made the choice to exclude two-point conversion attempts, so unfortunately this beauty wasn’t included:
Whew. Here are the results:
I was surprised to find that reverses and lateral run plays were so valuable! In theory, they’re like having a player twice as good as Tom Brady passing the ball. Since running plays aren’t usually as valuable as passing plays, I expected trick passes to have higher EPA. Trick passes are still valuable, but a lot of the EPA was driven by flea-flickers alone. In fact, only counting running back and WR passes (n=194), the EPA was 0.024 — worse than a normal pass! Flea-flickers, when only grouped by themselves (n=66), had an EPA of 0.3 — very valuable. I guess the risk of your non-QB throwing an interception is too high.
Of course, if a team fakes punts all the time then defenses will adjust and the EPA will go down. The appropriate criticism here is that the novelty of the play is what makes them valuable. Still, my analysis suggests that teams should definitely run more reverses and fake punts.
On a different note, I would argue that risk tolerance isn’t a good reason to avoid faking kicks or running reverses. Here’s an analogy. In blackjack, it feels uncomfortable to hit on 16. You frequently bust and feel stupid. However, it’s almost always the mathematically best play because the dealer will often beat 16. Often you’re simply giving the house extra edge when you stay on 16.
Like with hitting on 16, faking a punt on your own side of the field and failing probably feels stupid. However, it’s mathematically advantageous. There’s plenty of room to try it more than 1% of the time.
Similarly, I think it’s an important point that all of these trick plays have high standard deviation. That suggests that, even if the EPA of those plays is the same as a normal play, sometimes when a team is overmatched it’d still be advantageous to try them. If you’re up to an 80% win probability, game theory says don’t try a bunch of crazy plays that might immediately reverse that win probability. For the same reason, teams intelligently run more often when they are far ahead. On the flip side, if you’re sitting on a 30–40% win probability, a fake punt or field goal might be the type of play your team needs to jump to a 55% win probability. Let’s see what teams actually do:
Average Win Prob at time of punt attempt: 0.462Average Win Prob at time of fake punt attempt: 0.398Average Win Prob at time of field goal attempt: 0.578Average Win Prob at time of fake field goal attempt: 0.561Average Win Prob at time of run attempt: 0.559Average Win Prob at time of trick run attempt: 0.534Average Win Prob at time of pass attempt: 0.490Average Win Prob at time of trick pass attempt: 0.557
It seems that teams are only correctly using fake punts when they are in a tough situation. Field Goals, trick passes, and trick runs are all only being run while the teams are (on average) ahead already. That is suboptimal, and probably due to coaches being too conservative.
Of course, all this being said, there are certain trick plays that are never optimal to run:
This article has been kicking around in my brain ever since I saw an impressive Mohamed Sanu stat line. I’m going to be sure to give a shout out. On 8 attempts when Sanu passes the ball, he’s on average added 2 points a play! That’s insane! Here are those 8 passes:
(14:53) (Shotgun) Direct snap to M.Sanu.  M.Sanu pass deep middle to A.Green for 73 yards, TOUCHDOWN.(12:48) M.Sanu pass deep right to G.Bernard pushed ob at CLE 9 for 25 yards (B.Mingo). {Ball lateraled from Dalton to Sanu} Cleveland challenged the pass completion ruling, and the play was Upheld. (Timeout #2.)(1:03) M.Sanu pass deep right to B.Tate pushed ob at ATL 24 for 50 yards (D.Lowery).(:06) (Shotgun) M.Sanu pass short left to A.Dalton for 18 yards, TOUCHDOWN. {Dalton pitched out to Sanu, who then passed to Dalton.}(3:28) (Shotgun) M.Sanu pass short right to A.Green to TB 27 for 11 yards (B.McDougald). #14 lateral to #12(12:44) Direct snap to M.Sanu.  M.Sanu pass deep left to Ju.Jones for 51 yards, TOUCHDOWN.(8:56) T.Sambrailo reported in as eligible. Direct snap to M.Sanu.  M.Sanu pass incomplete deep left to J.Jones.(8:55) (Shotgun) M.Sanu pass short right to M.Ryan for 5 yards, TOUCHDOWN.
So, there are other variables at play. If one of your wide receivers has a golden deep ball then it makes the math easier. The video makes it even more impressive:
To conclude, NFL teams should run trick plays more often*. If nothing else, it’s fun. Code can be found here.
*depending on many factors