Full Self Driving Is Possible With Existing Tesla Hardware
By: Lewis Gavin
Tags: Tesla, Machine Learning, Technology, Data Science, Self Driving Cars

The above image and the one used at the top of this post were both taken from videos provided by @greentheonly (on Twitter), who has lots of videos from the different Tesla cameras on his YouTube channel.

The two images show what is captured by the left pillar camera and the front fisheye camera whilst the car is on Autopilot driving through Tokyo. The purpose of these images is to simulate what the cameras see when at a junction.

In the image at the top of this post, the car can clearly see down the street on the left identifying a pedestrian, the car and you can already see a lorry that is just starting to make its way into the image.

The same is true for the second image (just above). It can see down the street as much as you or I could when driving, certainly enough to make a decision on whether it is clear to continue.

Where things get interesting is when the view is obstructed as shown in the following image.

Tesla right pillar view obstructed (image source)

The right pillar camera can see some of the road to the right, but its view is obstructed by a car, pedestrians and the building. This same would be true for a human driver too.

For argument's sake, if this junction wasn't protected by traffic signals and the car would need to decide for itself if it is safe to proceed then how would it deal with it?

Well the FSD beta testers have already showcased such a scenario, where the car slowly creeps forward until it is able to see more and therefore make a more informed decision (as shown in the video below).

Now while this isn’t “proof” that the car can fully navigate each and every junction, what it does show is that the cameras are providing enough data that given enough training, the car should eventually be able to.

The reason I think this misconception exists about the car's visibility is that it’s rare to actually see footage from the B pillar cameras. Whether viewing the backup cameras in the vehicle or looking at sentry clips, you only see the front, rear and rear-facing side cameras, not the B pillar images. Because of this, it’s hard to imagine what they actually see and in some instances are probably forgotten about altogether.

So What Is Next for Full Self Driving?

New versions of the FSD beta are being delivered weekly each of which bringing great improvements according to the beta testers on Twitter.

When quizzed about when the beta will be released to a wider audience, Musk said that if next weeks release looks good (release on the week commencing 23rd November 2020) then they will widen the beta.

I don't believe this means a full rollout to all owners who have the FSD package in the states, just that it will be expanded to even more people than the select few that have it at the moment.

Elon has also confirmed that an end to end neural net that spits out steering, braking and acceleration as outputs from the camera feed is in development. Currently, the camera feed is fed through neural nets for perception but rules around acceleration are built using conventional code.

All of this has been confirmed possible with the current hardware. Again, it appears that the hardware isn't an issue at all when it comes to solving the FSD problem and that the team only needs to nail the software.

When thinking about this it makes sense. They have given the car a better field of view than a human with extra sensors too. This is fairly trivial to solve as we know and understand the field of view of humans when driving a vehicle. What we don’t understand is how the human brain translates that input into driving commands.

This is the big problem and the only thing holding FSD back, not the hardware.