
Amazon S3 Path Deprecation Plan – The Rest of the Story - jeffbarr
https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/
======
DVassallo
Thank you for listening!  The original plan was insane.  The new one is sane.  As
I pointed out here
[https://twitter.com/dvassallo/status/1125549694778691584](https://twitter.com/dvassallo/status/1125549694778691584)
thousands of printed books had references to V1 S3 URLs.  Breaking them would
have been a huge loss.  Thank you! 

~~~
inopinatus
If we're talking textbooks, well then.  This is a textbook case for the 301
HTTP response code. 

~~~
jasonkester
The old REST-style S3 URLs are specifically excluded from being able to
redirect:

[https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-
page-...](https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-
redirect.html)

You can create a new bucket or switch your existing one to "Static Website
Hosting" mode to enable the ability to 301 your content going forward.  But the
URL for the "website" version isn't the same as the REST URL.  And again,
there's no way to redirect from the old naming scheme to the new one. 

If you have content that you've ever linked with one of those URLs, it's stuck
there forever. 

~~~
codetrotter
> And again, there's no way to redirect from the old naming scheme to the new
> one. 

For customers, no.  For Amazon itself, yes.  And I think that is what parent
commenter meant.  That Amazon should 301 all requests that are using old paths. 

~~~
xchaotic
It's not that simple, unfortunately - it won't work for the old dotted
addresses and S3 is not HTTP

------
chipperyman573
Still doesn't help with domain censorship.  This was discussed in-depth in the
other thread from yesterday, but TLDR, it's a lot harder to block
[https://s3.amazonaws.com/tiananmen-square-
facts](https://s3.amazonaws.com/tiananmen-square-facts) than
[https://tiananmen-square-facts.s3.amazonaws.com](https://tiananmen-square-
facts.s3.amazonaws.com) because DNS lookups are made before HTTPS kicks in. 

~~~
btown
It actually helps tremendously, since at the very least now there can be a
"black market" for legacy (pre-Sept-2020) buckets, especially those on
dedicated accounts that can be provided to organizations spreading facts like
this. 

~~~
BrentOzar
> It actually helps tremendously, since at the very least now there can be a
> "black market" for legacy (pre-Sept-2020) buckets

Err, no, countries will just block the legacy bucket URL style and say that
only the bad guys would still be using it. 

~~~
shapov
That would mean they are blocking all S3 buckets indiscriminately. 

~~~
Ettvatre
Couldn't they just middle-man the traffic and block specific URLs? 

~~~
lugg
ssl prevents that. 

~~~
cthalupa
It explicitly does not.  It means there are additional barriers to doing it -
people would need to accept a bad cert (we already know the overwhelming
majority will), or they would need to slip in their own CA that allows them to
generate their own valid certs for MITM, but that is eminently doable for the
Chinese government inside of China.  They can then block all traffic for people
that do not use the cert that allows them to decrypt said traffic.  It
functionally is the exact same thing, and would still allow "legitimate"
traffic without problem. 

~~~
lugg
That's not what explicitly means.  Ssl explicitly does prevent mitm attacks
from intercepting URLs of requests. 

The fact you can get around it by ignoring the cert is a bit irrelevant.  It's
like saying locks don't work because people can break your window. 

~~~
cthalupa
As noted, you don't have to ignore the cert, and we're talking about state
level actors. 

And it's not the window.  It's like saying locks don't work if the state has a
master key, which they do. 

------
blaisio
This is interesting for a few reasons.  IMHO, the original deprecation plan was
reasonable.  Not generous, but reasonable.  Especially compared to what other
cloud providers (eg.  Google Cloud) have done.  It did seem like a diversion
from their normal practice of obsessively supporting old stuff for as long as
possible, but it really wasn't too bad. 

Responding to feedback, publicly, and explaining what they were trying to do
and why they needed to do it, is incredibly refreshing. 

This seems like a big PR win for AWS.  I'm left trusting and liking them more,
not less. 

~~~
DVassallo
How was the original plan “reasonable”?  S3’s FAQ talks about durability in
terms of tens of thousands of years
[https://aws.amazon.com/s3/faqs/](https://aws.amazon.com/s3/faqs/).  To honor
that claim, AWS has the burden to support v1 URLs until the end of the
internet. 

~~~
whoisjuan
Storage durability has nothing to do with this.  Changing how you access data
saved in storage is a reasonable change to keep up with the evolution of
networking technologies. 

The change here was deprecating an access pattern, not destroying data or
anything remotely similar. 

~~~
DVassallo
Tell that to the author of “The Peace Corps and Latin America”, who used S3 v1
URLs dozens of times in the book, with the assumption that they’d be available
forever:
[https://books.google.com/books?id=Q312DwAAQBAJ&pg=PA135&dq=h...](https://books.google.com/books?id=Q312DwAAQBAJ&pg=PA135&dq=https://s3.amazonaws.com&hl=en&sa=X&ved=0ahUKEwiA2tek3o3iAhWYHjQIHVOjDSwQ6AEIKjAB#v=onepage&q=https%3A%2F%2Fs3.amazonaws.com&f=false)

And thousands of other books like that. 

~~~
oconnore
Did they discuss that with Amazon prior to publishing?  That seems like a
completely silly assumption to make for a product that’s been around for only
13 years. 

~~~
DVassallo
Also every AWS book out there that has instructions on how to install the AWS
CLI uses the v1 URL!  You know why?  Because that’s the official URL! 
[https://twitter.com/dvassallo/status/1125502432924975104?s=2...](https://twitter.com/dvassallo/status/1125502432924975104?s=21)

------
luhn
> Bucket Names with Dots – It is important to note that bucket names with “.” 
> characters are perfectly valid for website hosting and other use cases. 
> However, there are some known issues with TLS and with SSL certificates.  We
> are hard at work on a plan to support virtual-host requests to these
> buckets, and will share the details well ahead of September 30, 2020. 

I’m mystified how they’re planning on doing this.  Anybody care to speculate? 

~~~
regecks
They're already a CA, could they reasonably just issue a certificate for every
bucket?  I have no idea how many buckets there are in total. 

~They probably couldn't take the Cloudflare approach of jamming 100 customer
domains onto each certificate, since that would leak bucket names too easily.~

~~~
RKearney
> They probably couldn't take the Cloudflare approach of jamming 100 customer
> domains onto each certificate, since that would leak bucket names too
> easily. 

Issuing one certificate at a time wouldn't make a difference since they're all
submitted to public CT logs.  Bucket names shouldn't contain sensitive
information and security through obscurity is a bad idea. 

~~~
blattimwind
> security through obscurity is a bad idea

Obscurity is a good and sensible layer for defense in depth.  Systems A and A'
were the only difference for A' is added obscurity will result in A' being
more difficult to attack. 

------
kenhwang
For anyone still confused to why AWS dominates the cloud market, it's because
they're willing to grandfather features with a reasonable sunset horizon. 

~~~
rsanek
I'd say the momentum from having no competition in the space for nearly a
decade is far more impactful. 

~~~
phh
I'd say it's a combination of both. 

Not having competition means they got a lot of users, but having long-term
feature support means people didn't run away. 

------
valgaze
Malloc for the internet: "We launched S3 in early 2006.  Jeff Bezos’ original
spec for S3 was very succinct – he wanted malloc (a key memory allocation
function for C programs) for the Internet.  From that starting point, S3 has
grown to the point where it now stores many trillions of objects and processes
millions of requests per second for them.  Over the intervening 13 years, we
have added many new storage options, features, and security controls to S3." 

------
raiyu
It's nice to see that instead of deprecation support for the old paths will
continue for all buckets created on or before the cut-off date of Sept 30,
2020. 

So if you don't want to change, you can continue using the old paths.  Just
might limit access to some new features coming later that are dependent on the
virtual host sub domains. 

------
ryanbigg
This is a great step forward.  Particularly changing the rules a little so that
old buckets won’t break after a certain date. 

Thank you for taking the time to write this up Jeff. 

~~~
jeffbarr
You are welcome.  And now, back to my stay-cation. 

------
bilater
Okay probably a dumb question but why can't they just have an automatic
redirect from the path style to the virtual hosted ones under the hood?  People
get both options up front while they can work with the one they like. 

------
nik736
"In this example, jbarr-public and jeffbarr-public are bucket names;
/images/ritchie_and_thompson_pdp11.jpeg and /jeffbarr-
public/classic_amazon_door_desk.png are object keys." 

I think this should be:

"In this example, jbarr-public and jeffbarr-public are bucket names;
/images/ritchie_and_thompson_pdp11.jpeg and /classic_amazon_door_desk.png are
object keys." 

~~~
jeffbarr
You are correct; thanks for spotting this!  All fixed. 

------
gundmc
Props to Amazon for listening to feedback and altering course. 

------
tanilama
Grandfathering is a good idea.  GJ AWS. 

~~~
andybak
"GJ"? 

~~~
rekshaw
Good job

------
el_benhameen
Kind of tangential, but is Bezos a programmer type?  I thought he came from
banking or the big 4.  I’m curious if the “malloc for the internet” bit is
verbatim. 

~~~
hbosch
I don't think Jeff is a programmer, but I think he is very smart and his
success shows that he can understand and create businesses around lots of
different types of concepts and ideas.  I imagine in reality there is some
amount of collaboration that happens before Bezos spouts off something like
"create a malloc for the internet", but in any case, it's very strong for his
brand of leadership that the lore states it came straight from him. 

~~~
fgonzag
1.)  He graduated from CS at Princeton. 

2.)  Only an actual programmer would know what malloc is and how it works (he
knows what it is exactly since he has the idea "malloc for the web")

~~~
hbosch
Point taken, I didn’t know Bezos had an engineering background!  Amazing. 

------
ZiiS
It seems to me that adding a 301 redirect from the old URL to the new would
not unresonably stress the resources of AWS?  It seems perfectly resonable to
update the library access, but breaking old URLs seems unessesary.  They could
even add a second of latency to incentivise people who can update their links. 

~~~
joemag
Some HTTP clients don’t support redirects, or at least require an explicit
configuration to enable them.  So this would still be a breaking change for
some applications. 

------
gigatexal
Yeah pushing the new way is fine but not removing the logic to resolve the old
way is better. 

~~~
jasonhansel
Agreed!  I think Amazon definitely made the right call here. 

------
freeasindave
Pre-signed urls still come back from the S3 SDK as a V1 path style.  I'm
assuming this either changes at some point, or that will continue to work? 

------
nullecksor
This was a DOA when I first read it.  S3 or AWS wouldn't break a single
customer before changing anything. 

------
ahmhn
Does this change affect S3 access via the various AWS SDKs, or just the format
of URLs? 

------
parliament32
I still don't get why there was such an uproar about this: Amazon should just
issue a "301 Moved Permanently" and be done with it. 

If your app for some arcane reason doesn't understand an HTTP status code
that's been around for 20 years... your code is bad and you should feel bad.