how does google not run out of space - Page 1
Forum section: electronics / general

author: sony mavica
author_rank: Frequent Contributor
timestamp: April 26, 2017, 01:42:06 am
content: i have always wondered this as the amount of data sent to the google severs ever couple of seconds is like 1tbplus there sites like YouTube have like 6 different encodes of the same video and up to 9 if original video is 8k and every video uploaded to youtube they will keep the original filehow can they keep up with so much data being sent to there servers

author: Ampera
author_rank: Super Contributor
timestamp: April 26, 2017, 01:45:57 am
content: Google is unimaginably rich. They can afford that space. But that is interesting, what sort of drives do they use, if there are any special storage devices, etc.

author: RoGeorge
author_rank: Super Contributor
timestamp: April 26, 2017, 01:49:04 am
content: One of the Google's long term missions is to be the repository for all human produced information.Google is constantly increasing/upgrading their storage space, and there are many data centers around the world:

author: tautech
author_rank: Super Contributor
timestamp: April 26, 2017, 01:49:22 am
content: Which begs the question; what form of data storage has the highest density ?Used to be tape.

author: Dubbie
author_rank: Unknown
timestamp: April 26, 2017, 01:59:21 am
content: Would be flash memory probably.Not the cheapest though!

author: helius
author_rank: Super Contributor
timestamp: April 26, 2017, 02:02:57 am
content: I think atomic force microscopy still has the record for densest information storage.Now reading back the data, that's a bit more complex

author: sony mavica
author_rank: Frequent Contributor
timestamp: April 26, 2017, 03:18:13 am
content: Quote from: cleaningOut on April 26, 2017, 02:06:36 ambut like mini dv does the tape degrade quickly after a couple of uses

author: Red Squirrel
author_rank: Super Contributor
timestamp: April 26, 2017, 03:25:20 am
content: Since they own their own data centres they can pretty much control every aspect of it. My guess is they have dedicated staff that literally all they do is add more disk space. Like order/assemble/rack/plug in systems. They probably have custom setups where they snap everything together, put in like 60 of the highest TB drives available in a single 4U chassis, and slide it in, plug it in, then the cloud system takes over from there. probably have like 10+ people who do this non stop all day. When they run out of floor space they build a new data centre. At some point they probably take down older storage servers to replace with newer higher density drives, and just repeat.But it is mind boggling just how much data they do produce so I do wonder how they do in fact keep up, it seems even if people are putting together storage pods day after day it would still not be fast enough.

author: digsys
author_rank: Unknown
timestamp: April 26, 2017, 03:48:28 am
content: Quote from: sony mavicaActually, I worked with a mate who looked after a data storage centre. The tapes can last a very long time, 10-20 yrs, depending on quality BUT, to achieve that -they are kept in a temp controlled room, flipped every few months or so to avoid print-through, spooled and re-spooled. Orientation is very important, due to effects of gravity.

author: Red Squirrel
author_rank: Super Contributor
timestamp: April 26, 2017, 04:16:47 am
content: I had looked into tapes for my home backups but felt they are not really practical due to those issues. You really need to cycle them often to avoid bit rot, and each time you do cycle them, you reduce their life. They are only good for a couple thousand spools. I suppose if they are kept in a faraday cage at like -100C they might last longer. But print through might still be an issue.

author: Brumby
author_rank: Unknown
timestamp: April 26, 2017, 04:58:44 am
content: While the hardware question is a good one - you have to admire the software design that manages it all. This is one place where I imagine scalability is challenged on a daily basis.

author: digsys
author_rank: Unknown
timestamp: April 26, 2017, 05:00:30 am
content: Quote from: Red SquirrelThis may actually be a good school / science experiment for the space station !! They're always looking for different ideas. Send up 20 tapes, with 20 control tapes on Earth, then bring 1 tapedown every year or so, and test against the reference ones. Any kids listening ? :-)

author: CJay
author_rank: Super Contributor
timestamp: April 26, 2017, 06:19:11 am
content: I have it in my head that Google use commodity drives for their storage, the kind of thing you'd find in a desktop PC but that's based on a memory of a hardware reliability study they published a few years ago.They also used to run their own blend of server which amounted to little more than a commodity board on a folded metal tray, I think they'd worked out for the number of cores and terabytes needed that by having massive redundancy in their server and disk arrays and using consumer grade parts (albeit mid to high end ones) instead of dedicated server hardware they would save a *lot* of money at that the cost of a shorter lifecycle (however I do remember the reliability difference between enterprise and commodity drives being surprisingly small after early mortalities were weeded out and if they were all treated well)It may of course have all changed now and they've decided that racks crammed full of flash chips are much better.I think this is the study or a close relative:http://matrix.lt/download/google-hdd-disk-failures-analysis.white_paper.pdf

author: Red Squirrel
author_rank: Super Contributor
timestamp: April 26, 2017, 07:48:37 am
content: Yeah the slight performance difference between "enterprise" and consumer drive is probably too negligent to justify the increased cost. And individual hard drive is also seen as a very tiny sector as far as Google is concerned - heck, even me. I don't store data on single hard drives, I store it on multiple at once. AKA RAID. I imagine Google has their own custom raid system that can span a crazy amount of drives with a crazy amount of performance and data safety tweaks.Due to the high volume of writes flash is probably also not viable as flash has write limitations so I imagine they'll always stay on spinning disks, maybe also ram.

author: Kilrah
author_rank: Unknown
timestamp: April 26, 2017, 07:54:27 am
content: Quote from: sony mavica on April 26, 2017, 01:42:06 amThey add more, and more, and more, all the time.

author: rrinker
author_rank: Super Contributor
timestamp: April 26, 2017, 01:41:01 pm
content: A common commercial SAN uses 3U disk trays that hold 15 drives. There are now 12TB enterprise class drives. That's 180TB for every 3U, and you get 12 disk trays plus the 4U storage controller in a rack (with a couple of U left for the switch). 2 PB per rack.And there are other brands and models of storage systems that have a much greater density.

author: boffin
author_rank: Unknown
timestamp: April 26, 2017, 02:12:51 pm
content: If you're looking for 'do it yourself' storage solutions, these are two pretty good places to start. I've used numerous 6048Rs over the last few years, and they're pretty damn rugged and easy to deal with. 36 drive version is pretty annoying if the rack isn't well organized to get those back drives out, but the 24drive version is really nice and simple.https://www.supermicro.com/products/system/4U/6048/SSG-6048R-E1CR36N.cfmBackblaze sell silly cheap backup (and block storage), and they opensourced made their own, details are here. Problem is it's impossible to replace drives 1 by 1 when they break. Lots of history about their pods on their website, and here's the most recent.https://www.backblaze.com/blog/open-source-data-storage-server/

author: mleyden
author_rank: Contributor
timestamp: April 26, 2017, 04:20:03 pm
content: Having spent some time working on a Google for Work rollout last year...Google will store your data in at least 3 and up to a max of 5 of their datacenters. You will access data from the datacenter your are nearest to. I assume it is similar for all their services... So take the 1TB / sec and it becomes 3TB / sec! Very few can compete with them - perhaps Amazon and Microsoft - but noone else could afford the infrastructure.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: April 26, 2017, 04:28:34 pm
content: Quote from: Red Squirrel on April 26, 2017, 07:48:37 amThat would assume they actually delete things. Flash write limitation only comes into play when you write, delete, write, delete, write, delete, over and over again, hundreds of TB per drive. If all you're doing is archiving, that's one write cycle, and then it just sits there forever being occasionally read from.

author: rdl
author_rank: Super Contributor
timestamp: April 26, 2017, 04:34:31 pm
content: I doubt Google ever deletes anything.However, would it actually require that much writing? Deleting is not usually the same as erasing. Isn't the data is just marked as over-writable? I suppose that might depend on the system though.

author: rob77
author_rank: Super Contributor
timestamp: April 26, 2017, 04:40:24 pm
content: Quote from: rrinker on April 26, 2017, 01:41:01 pmwhat you mean by common commercial SAN uses 3U .... ?SAN = storage area network it consists of SAN switches/directors , storage arrays, hosts and all is connected through fibre channel (scsi commands encapsulated into FC protocol transported over fiber)....the SAN directors we use have hundreds of 8Gbit/s FC ports, storage arrays we use are a full cabinet/rack (or two) with hundreds of spindles (disks) and the hosts are connected through FC host bus adapters with at least 2 ports. a common commercial SAN has 2 fabrics (each fabric is a separate FC network) for redundancy so you have at least 2 switches/directors and both storage arrays and hosts are connect to both of them (half of the available FC ports per fabric)...so a common commercial SAN is definitely not a 3U disk array.

author: sokoloff
author_rank: Super Contributor
timestamp: April 26, 2017, 04:46:48 pm
content: Quote from: rob77 on April 26, 2017, 04:40:24 pm@rrinker specifically said uses 3Udisk trays. IME, that's a very common (and perhaps the most common) form factor for the disk trays.He was clearly trying to Fermi estimate the amount of storage per rack footprint.

author: rob77
author_rank: Super Contributor
timestamp: April 26, 2017, 04:57:51 pm
content: Quote from: sokoloff on April 26, 2017, 04:46:48 pmmy point is that a disk array is not a SAN , it's much more complex than that..... so a SAN is not using disk trays. the storage array connecved to a SAN is using disk trays with disks... and some of them might be using 3U trays..

author: Kilrah
author_rank: Unknown
timestamp: April 26, 2017, 05:56:34 pm
content: That's exactly what he said in the rest of the sentence, that the SAN would be the rack with multiple trays, controller and switches...Quote from: rdl on April 26, 2017, 04:34:31 pmYes but the whole point of deleting is to make room to write something on there again... so it will eventually lead to a write cycle.

author: bitseeker
author_rank: Super Contributor
timestamp: April 26, 2017, 07:18:18 pm
content: Quote from: CJay on April 26, 2017, 06:19:11 amYes, that's the one that came to my mind as well, as I read the OP.

author: Red Squirrel
author_rank: Super Contributor
timestamp: April 27, 2017, 02:51:03 am
content: Quote from: suicidaleggroll on April 26, 2017, 04:28:34 pmTrue I guess when you make a change it won't overwrite it will just create a new copy. I now wonder, perhaps they don't delete anything either, like youtube videos. So that would mean they REALLY need to keep expanding storage at a crazy rate.

author: fubar.gr
author_rank: Unknown
timestamp: April 27, 2017, 10:26:32 pm
content: There are several tricks used in the industry to deal with minimizing hard drive space.One of them is thin provisioning. For example, there are 1000 gmail accounts, and each one has a 5GB limit.So in theory google would need a total of 5000GB hard disk space. But in practice, many of these accounts are dormant and most users will never come anywhere near the 5GB space limit. So in reality you can only have a fraction of the total advertised space available, with some headroom for safety and then add more drives if needed.Another trick is data dedublication. For example, I send a file attachment from my gmail account to another. Now we both have this file in our gmail accounts. But is is the same freaking file! It doesn't make sense to keep two different instances of the same file. Gmail only stores it once and all links lead to this one file.But even with these tricks google still needs a shitload of hard drive space, so they keep adding more datacenters to their network.

author: bill.coghill
author_rank: Contributor
timestamp: April 28, 2017, 08:19:22 am
content: I can tell you they use a lot of mechanical drives and have people who destroy them using a fantastic big spiky tool once they reach a certain age. They don't like SSDs as they can be hard to fully destroy to stop data leakage. Longer term storage is on tape with robots for tape handling.Sent from my Nexus 5X using Tapatalk

author: AndyC_772
author_rank: Super Contributor
timestamp: April 28, 2017, 08:42:35 am
content: Quote from: Red Squirrel on April 26, 2017, 07:48:37 amI don't think that's the case. I recall a request to the Flash industry (from Facebook, IIRC), to develop a cost-reduced Flash technology, which more-or-less simplified the endurance requirement to 'write once, read never'.Think about it... once something is posted to the web, it's rarely changed. A photo of your dog gets posted, viewed by the two or three people in the world who care about it, and then it's forgotten completely (but remains on the server). Physically, it sits in a cache for the first few days after being uploaded, then it needs to be moved somewhere for long-term storage. It may never be accessed again, and it certainly won't be getting updated on a regular basis.Ultra low cost, very low write endurance, but nevertheless long-term reliable storage is what's needed.

author: CJay
author_rank: Super Contributor
timestamp: April 28, 2017, 09:53:52 am
content: Quote from: fubar.gr on April 27, 2017, 10:26:32 pmDe-dupe and thin are two of the most common, thin being very cost effective if monitored and managed properly, de-dupe, I don't know how that'd scale across Google's geographically diverse datacentres and massive storage needs, I can imagine some measurable performance hit unless the algorithms took into account the location of users which seems rather complex?

author: Hypernova
author_rank: Unknown
timestamp: April 28, 2017, 01:23:20 pm
content: Quote from: suicidaleggroll on April 26, 2017, 04:28:34 pmThe charges in high density triple layer flash cells dissipate in time frame measured by the low thousand hours. They need to be cycled around and that does impose a limit on their lifespan. As cold storage they are worse than tapes.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: April 28, 2017, 02:04:46 pm
content: Quote from: Hypernova on April 28, 2017, 01:23:20 pmThere is a lot of FUD in this post...First, the "low thousand hour" charge dissipation time is for:1) Unpowered drives, it doesn't apply when the drive has power2) Only applies to drives that have already hit and exceeded their write cycle limit3) Only applies to drives being stored at extreme temperaturesYes, if you take an SSD, abuse the hell out of it for years and manage to hit the write limit, then remove it from the computer and throw it in a box, and then put that box on a shelf in a 50C warehouse, the data will likely decay and become unusable after a few weeks/months. That's a pretty extreme use case though.I don't know what you mean by "cycling around" the drives reducing their lifespan...all you need to do is power on the drive to get around this decay clock, you don't need to write anything, you don't need to move anything. And who said anything about using flash media for "cold storage" anyway? I don't think anybody has suggested Google would archive PBs onto SSDs and then throw them in a box on a shelf in a warehouse for years, it wouldn't make any sense.

author: Jeroen3
author_rank: Super Contributor
timestamp: April 28, 2017, 02:20:59 pm
content: Quote from: sony mavica on April 26, 2017, 01:42:06 amHigh level file systems. Likegluster.Simple way to get a petabyte at your small home office. You know, to store... stuff....You can also look-up LTT's Petabyte project.

author: kc8apf
author_rank: Regular Contributor
timestamp: April 29, 2017, 10:30:24 pm
content: Hi there. I managed Google's hard drive team for a year and now manage their server management controller firmware teams. Google uses a lot of enterprise-class hard drives, a lot of flash, and a lot of tape. Hard drives are for bulk storage, flash is when access speed is critical, and tape is for backups. Cluster-level filesystems (descendent of GFS, similar to gluster) hide the details of individual storage devices from the apps and provide fault tolerance.Google has been getting more involved in Open Compute Project. If you're interested in what a Google server looks like, the schematics and board layouts for a recent design, named Zaius, are available on GitHub athttps://github.com/opencomputeproject/zaius-barreleye-g2.Sent from my Nexus 5X using Tapatalk

author: John_ITIC
author_rank: Frequent Contributor
timestamp: May 01, 2017, 01:04:57 am
content: Quote from: tautech on April 26, 2017, 01:49:22 amMagnetic media (hard disks). And it will keep being many times cheaper / Mbyte than solid state memory for 15 years.Source: Western Digital (manufacturer of both solid state and HDDs).

author: Kilrah
author_rank: Unknown
timestamp: May 01, 2017, 06:17:34 am
content: Quote from: John_ITIC on May 01, 2017, 01:04:57 amWho wants to make sure any trend they attempt to set allows for comfortably amortizing their HDD manufacturing plants, while they hope competition doesn't require them to invest in more SSD manufacturing too quickly

author: Jeroen3
author_rank: Super Contributor
timestamp: May 01, 2017, 07:10:32 am
content: Quote from: tautech on April 26, 2017, 01:49:22 amI think that flash will win in terms of absolute density.But there are three factors involved in choosing storage:- Density.- Price.- Latency.Can't have all 3.Flash has density and latency. Tape has density and price. HDD's have price and latency.

author: cncjerry
author_rank: Unknown
timestamp: May 07, 2017, 06:19:51 pm
content: I've been in the IT business since '76, retired two weeks ago. I had clients that have 65PB spinning which is nothing compared to large content providers and insurance companies. Most use tape. It's all automatic and cached. For example, if you stream a video from the beginning you don't notice the delay because the look-ahead grabs and spools the tape before you get to it. If you try jumping ahead in a video stream that's when you notice the lag. I have friends that sold storage arrays to Google and I could find out, but I doubt they have 100PB spinning. They have a lot of flash now for tier 0. Remember, they are more of an indexing and caching system though they have content now.If you want to see storage, look at large banks and insurance companies. The largest I've seen in my career was a large insurance company who's data center was built into bedrock that could withstand a 20Mt blast in NY city as they were about 70 air miles away. I heard they had 135PB spinning but were reducing and going to all flash arrays because with compression and de-dup, the economics were in favor of SS arrays.

author: Red Squirrel
author_rank: Super Contributor
timestamp: May 07, 2017, 07:58:38 pm
content: Downside in this day and age is that because we store everything digitally, it's going to completely vanish one day, there won't be a single record.Maybe that's a good thing though, do we really want generations in centuries from now realizing that we just spent most of our time looking at cat videos and making memes?

author: Kilrah
author_rank: Unknown
timestamp: May 07, 2017, 09:01:26 pm
content: Why?I believe the opposite, we're finally able to store data with no alteration for any length of time.