If 10 million people are watching youtube does that mean there are 10m harddrive - Page 1
Forum section: electronics / general

author: Beamin
author_rank: Super Contributor
timestamp: May 09, 2019, 08:33:52 am
content: When you watch a video on youtube is there a actual hard drive at the server end that spins up seeks the video on the disk then plays back that video with that particular hard drive staying in use by you until the video is over? How do they do this if say 100 million people are watching it at once they don't have millions of hard drives or do they? And if 65 years of videos are uploaded each day how do they keep up with the memory requirements? How many gb/tb are in each sever farm? It seems like they would end up spending huge amounts of money to store videos many of which are shit and will get a handful of views each year and really just take up memory and cause delays in indexing all these videos.

author: martin1454
author_rank: Regular Contributor
timestamp: May 09, 2019, 09:03:42 am
content: Well, there is not 100 million harddives for 100 million people - 1 harddrive can read multiple videos off to multiple users, so it is probaly something like 1 HDD = 10-100 users.

author: DannyTheGhost
author_rank: Contributor
timestamp: May 09, 2019, 09:11:16 am
content: Well, i somewhere read article about cloud servers holding dataFirst of all, who said that 100 million people will watch 100 million different videos?So, for comfort reasons, data which is 'popular' (that is frequently requested) will be moved from high-capacity low-speed hard drives to low-capacity high-speed drives so more people will accept it fastIn case when data is stored for long time it will be moved back to those 10TB+ hard drivesStorage is not the problem when we talk about servers - it's its bandwidth

author: AndyC_772
author_rank: Super Contributor
timestamp: May 09, 2019, 09:12:21 am
content: Popular videos will be held on SSDs, or even in a RAM cache, for speed and power reasons. They'll only get moved onto hard drives once demand has fallen.

author: EEVblog
author_rank: Unknown
timestamp: May 09, 2019, 09:14:36 am
content: Quote from: Beamin on May 09, 2019, 08:33:52 amYes, and no.The most viewed videos would get priority space and caching at the local CDN's

author: Electro Detective
author_rank: Super Contributor
timestamp: May 09, 2019, 09:44:06 am
content: One should ask about all the server wastage on the amount of garbage commercials/mega annoyances on Youtube,stuff no normal consumer would have use for or ever buy,much less understand wtf it iscommercials clearly produced and performed by hoards of insane asylum escapees and or ice indulging coffeeholics.and then there's the annoying commercials that must be paying Youtube a fortune to bully and jack our browsers, like WIX, Grammarly, UAP, and others,BS that can easily pollute a short video with heaps of ads, popups and browser freeze ups and crashes.I suspect the problem is frustrated viewers using ad blockers,and Youtube kicking back harder, which hits the legit 'cop it sweet' non ad block users like me

author: bloguetronica
author_rank: Frequent Contributor
timestamp: May 09, 2019, 10:00:53 am
content: Quote from: Electro Detective on May 09, 2019, 09:44:06 amThe fault is on YouTube and not on the ad-block users. Ad-block users are as legal as any other users, since they don't wan't their PC's plagued by virus. On my smartphone, google ads often suggests me ads to spyware, bloatware or virus on legit websites. On the PC, ads are no less dubious.Kind regards, Samuel Lourenço

author: GlennSprigg
author_rank: Super Contributor
timestamp: May 09, 2019, 10:59:00 am
content: NOT just speaking of 'Youtube', but in general....A lot of lay people don't grasp how big 'Google' is !! They imagine a 'server/servers' somewhere,when there are huge buildings all over the globe, each filled withthousandsof computers/servers !!Not to mention Google's ridiculously huge & complexarray of Databases, as a result of being, (in avery simplistic analogy!), like a huge global 'Index' for 'Yellow-Pages' phone-books that was missing.It still amazes me when I do a 'Search' for a phrase/image etc, & finds3,600,000 resultsin say.3 Sec!!!(Although yes, that includes smart analyzing/preempting of what you type, WHILE you are typing it).Of course some of their data is updated almost in real time, or up to 24 hrs later to update new DomainName Server info for new sites etc, but 'generally', (I think??), their Systems stillre-downloadtheENTIREWorlds Internet, at least once a week, to update all their servers/databases !!!No more "10 386-PC's daisy chained together in a shed" !!!

author: tooki
author_rank: Super Contributor
timestamp: May 09, 2019, 02:01:28 pm
content: Yep. Google’s infrastructure is incredible.The stats for YouTube alone are amazing. This was a Facebook post I made back in December 2017:QuoteThe last item just blows my mind. The storage (redundant and distributed worldwide!), the processing for re-encoding...

author: tooki
author_rank: Super Contributor
timestamp: May 09, 2019, 10:02:39 pm
content: Also, do you guys remember this classic???https://youtu.be/SXmv8quf_xMI want some of that “tracer T” action...

author: GlennSprigg
author_rank: Super Contributor
timestamp: May 11, 2019, 10:57:43 am
content: 'tooki' Ha !!!Luckily his video was made when he was an ancient 12 year-old or something.As when he was only 8 years old, he knew NOTHING man!!!!!!!!

author: tooki
author_rank: Super Contributor
timestamp: May 11, 2019, 07:33:58 pm
content: As the comments said... if that's the "next gen" of hackers (see his username), then IT security folks will soon be out of business!

author: james_s
author_rank: Super Contributor
timestamp: May 11, 2019, 08:01:13 pm
content: Quote from: Electro Detective on May 09, 2019, 09:44:06 amYouTube is unusable without an ad-blocker, I don't even bother to try.

author: mzzj
author_rank: Super Contributor
timestamp: May 11, 2019, 08:05:15 pm
content: Quote from: GlennSprigg on May 09, 2019, 10:59:00 amBack in the days I remember reading from somewhere that either google or Altavista used to transfer the search index (on tapes?) weekly across the atlantic on aeroplane as internet capacity at that time was too limited..

author: TomS_
author_rank: Frequent Contributor
timestamp: May 13, 2019, 07:19:04 am
content: Quote from: james_s on May 11, 2019, 08:01:13 pmI wouldnt say its unusable, thats a bit too far. Its perfectly usable.There are types of ads which I find particularly annoying though, namely the ones that are scheduled in to the middle of a video. Sometimes multiples of.Im quite happy watching ads at the start of a video, even if you cant skip them, if it means the rest of the video is uninterrupted - I mean, they have to make money to keep providing that service somehow. But interrupt the video and you wreck the experience. Sometimes, very well made, high quality content videos are ruined by multiple ads playing throughout.

author: LapTop006
author_rank: Unknown
timestamp: May 13, 2019, 09:20:46 am
content: Quote from: TomS_ on May 13, 2019, 07:19:04 amWhereas I simply pay the money for YouTube Red/Premium/Whatever it's called this week. The only times I see ads on YouTube is the about once I year I have to re-login, or in an incognito tab.Also, as I've mentioned elsewhere, I work for Google, for those who want to know more I suggest theborg paper.

author: mariush
author_rank: Super Contributor
timestamp: May 13, 2019, 09:51:38 am
content: To answer the original question...Google has multiple datacenters (places on the planet where they have servers).Youtube uses servers located in most (if not all) of Google's datacenters.Some servers are only used to process database requests.Some may not have hard drives but instead have loads of RAM, like for example 256-512 GB of RAM. For example, such servers would be used to serve the thumbnails, video previews, user logos, static close captioning ... stuff that doesn't change often.Instead of requesting a small file from a spinning disc, with all the latency and IO involved, one of these servers with lots of RAM keeps copies of these small files in RAM and serves them directly from RAM.Some servers may serve videos but not actually have the videos physically stored on them ... think of a big NAS with hundreds of hard drives, which is connected to the datacenter network with multiple 40-100gbps network cardsA single dedicated server with a 10gbps network card can probably only serve around 50-100 videos at any single time, in order to guarantee let's say minimum 10-30 mbps download speed - so when you hit play on a video, the server asks the big nas machine for the video, the nas machine sends it to the server at 10 gbps speeds and the server holds it in RAM and then sends it to the user at a consistent speed.Google saves the original videos and also creates multiple versions of the video at various resolutions and with various codecs, to support various devices (apple portables, various TVs etc).Depending on popularity of a video, it will be stored in more datacenters... if the video is not so popular, you may find out it's downloaded from a region that's further away from you (across an ocean for example)... same if the video is very old and accessed rarely... they'll gradually drop such videos from some datacenters, or reduce the number of resolutions available in datacenters with fewer servers (ex max 720p instead of 1080p or 4k)

author: rstofer
author_rank: Super Contributor
timestamp: May 13, 2019, 03:04:06 pm
content: A 1080p YouTube video will transfer about 50 MB per minute. Call it 400 Mb/minute and realize that I can download at 350 Mb/SECOND. In other words, they can blast the entire 60 minute movie to my PC in about 68 seconds. The duty cycle is about 1 in 60. 1 second of transfer for 10 seconds of movie - at 1080p.At 144p the transfer is about 1.3 MB per minute. A 60 minute movie would be 78 MB or 624 Mb and I could slurp that up in a couple of seconds.Internet bandwidth might be a limitation and certainly individual download rates will impact what is displayed (is dial-up dead?) but it is always the receiver that is throttling the transfer. Yes, the server will throttle, particularly when loaded but more that likely, throttling occurs on my end or on the small fiber optic coming to the neighborhood.My 350 Mb transfer rate is miniscule compared to the large pipes leaving the server farms.SSDs are up in the GB/second range for server applications. That 1080p movie will take, at most, 3 seconds of SSD drive access. A single drive can handle about 1200 such movies concurrently. For the 144p movie, the drive time would be a couple of dozen milliseconds and a single drive can deliver over 100,000 videos concurrently.Bandwidth is everything! That's why the ISPs want these huge data sources to pay more toward building up the infrastructure.ETA: All number are 'more or less'. The details aren't as important as the capabilities of large pipes and large SSDs. RAM is faster, HDDs are slower (a lot slower) but under any condition, it isn't one drive per movie.

author: rstofer
author_rank: Super Contributor
timestamp: May 13, 2019, 03:09:11 pm
content: Quote from: mzzj on May 11, 2019, 08:05:15 pmNever underestimate the bandwidth of a station wagon full of floppies.

author: GlennSprigg
author_rank: Super Contributor
timestamp: May 14, 2019, 11:47:51 am
content: Quote from: rstofer on May 13, 2019, 03:09:11 pmYea... that's amazing. Speaking of 'floppies', I remember only so many years back, when Computer Magazineshas a single 1.44 mb disk taped to the cover. Was so excited to see what it contained !!!And the 'excitement' at hearing a '300-baud' 'modem' scream/blip/screech upon dial-up, was too much !But that was when real tight coding was done due to RAM limitations etc. An 'App' then, maybe '48k', did theequivalent to a lot of apps today from 10mb to 80mb. I know how it 'works' today, and why... but that accountsfor 95% of the 'data' on the 'net' these days.

author: Richard Crowley
author_rank: Super Contributor
timestamp: May 14, 2019, 12:14:03 pm
content: Over here in the Oregon Territories, Google takes advantage of the power from one of the Columbia River dams just a couple miles upstream, AND the cold water from the river to keep the servers cool. AND the location of a major optical fiber link in the interweb for connectivity. One of the Google server farms in The Dalles, Oregon:https://goo.gl/maps/NawMBTtp6GVcPETK9

author: Berni
author_rank: Super Contributor
timestamp: May 14, 2019, 01:59:07 pm
content: Yep its taken care of by CDN servers all over ths world.Infact to get around the bandwith issue Google offers free CDN servers to the internet providers. They give the ISP a few big rack mount machines that they can stick in there own racks and connect them into there own switches. These mini CDN servers fetch the popular content from Google and keeps it cached localy. Then when a internet subscriber wants to watch any of this content it gets intercepted and served to the user directly trough the ISPs switch. The data doesnt even go trough the internet since it never needs to exit the ISPs own network.The ISP is happy because it saves bandwith on there expensive terrabit links to the internet exhanges, so they will gladly swallow the cost of powering and cooling these small servers. Google is also happy becuase it saves bandwith on there own expensive internet links, saves server load and saves them on the power bill, all for the cost of a one time investment of handing out free servers.Something similar is done on large LAN partys. There people download games of tens of GB in size from Steam. So to keep things fast they build there own Steam cache server and plug it into there massive LAN. All traffic from the Steam download CDN is passed trough a bank of these cache servers. They hold on to the data and serve it from memory when needed wouthout going into the internet at all. Because many people are downloading tge same game this saves a ton of internet bandwith and allows for somthing like 100 people downloading a game at 1Gbit/s EACH.

author: james_s
author_rank: Super Contributor
timestamp: May 14, 2019, 05:23:25 pm
content: Quote from: rstofer on May 13, 2019, 03:09:11 pmHmm let's see. My station wagon is a 1990 so right around the start of the 3.5" floppy era has a stated maximum load capacity of 1,040lbs. Subtract ~180lbs for the driver and that leaves 860lbs for cargo. A quick search came up with 19g for the weight of a typical 3.5" floppy, so 860lbs is about 390kg and let's figure about 5kg for boxes to carry the disks in so 385kg, I get about 20,263 disks for a little more than 29GB of data assuming the disks are all neatly filled.I suppose that's not bad throughput even for a drive of several hours although it ignores the time it takes to transfer the data to and from the disks and the latency is wicked.

author: Berni
author_rank: Super Contributor
timestamp: May 14, 2019, 05:46:58 pm
content: Well back in the era of 3.5" floppies the best internet connection you could have likely gotten access to at home is a ISDN line. With the blazing 128Kbit/s of using both channels combined this would transfer the same 29GB of data in about 21 days.If you needed to transfer this data particularly far. Say from Alaska to Argentina (Practically to the the other side of the world) you would need to drive pretty fast to beat ISDN. One guy did this route as a road trip and documented it taking 23 days.But if you didn't have ISDN and had to do it with 56K dialup modems it would take at least 3 times as long so it would now take about 1/4 of a year to complete. Dialing directly would probably also cost more than actually driving your truck there.But today we have tiny SD cards that can hold >100GB so you can still use your truck to race against internet available to mere mortals and win.

author: Koen
author_rank: Frequent Contributor
timestamp: May 14, 2019, 05:53:18 pm
content: Quote from: mariush on May 13, 2019, 09:51:38 amI left the game a while ago so this quote led me to search where we are now and if anyone's interested : 24TB at SuperMicro servers, 12TB at Amazon EC2, 2TB at OVH rentals.

author: mariush
author_rank: Super Contributor
timestamp: May 14, 2019, 06:07:09 pm
content: Quote from: Berni on May 14, 2019, 05:46:58 pmYou could get in a plane with a backpack full of 2-4 GB drives or a bunch of tapes... maybe 2 days max with mixing with trains and ferries and cars...You could even argue it's cheaper depending how much in advance you could reserve tickets for cheaper price.Here in Romania when dial-up was still a thing and isdn barely started, isdn was... probably 200-300$ a month in today's money.You could bribe a train driver or the guy that checks tickets on train with 5-10$ to take a package and deliver it in hands of guy waiting at station...6-8h and your package is across country (~300-400 miles)Parents do this for university students even now... cheaper than regular mail and faster.

author: mariush
author_rank: Super Contributor
timestamp: May 14, 2019, 06:11:54 pm
content: Quote from: Koen on May 14, 2019, 05:53:18 pmI wasn't saying maximums, i know more is possible.256-512GB is the "sweetspot", doable with 8-16 x 32-64GB sticks, which aren't so expensive.for some uses it's better to have 50 servers with 128-512 gb instead of 2 with 12TB ...load balancing, network throughput, other reasons

author: Berni
author_rank: Super Contributor
timestamp: May 14, 2019, 06:49:53 pm
content: Yep backpack full of drives on a plane is certainly faster, but a van could also carry a lot more hard drives.They actually did race ADSL against a pigeon with a 4GB memory card strapped to it over a 60 mile route:https://phys.org/news/2009-09-carrier-pigeon-faster-broadband-internet.htmlEven in the modern day the postal service is used as a replacement for internet. Radio telescopes that collect vast amount of data send the results back in the form or mailing full hard drives. Because of how much data they have to get across this turned out to be faster and more economical than actually sending it over the internet.But internet has certainly improved a lot. I now have a symmetrical 150Mbit line at home and i barely use up any of the full bandwidth. If i did use all of it for download then i would fill up my 12TB NAS in about a week.If you get your hands on 1Gbit internet (Some places in the world have this for a non insane price) its possible to port scan every IPv4 IP address from 0.0.0.0 to 255.255.255.255 on the internet in 45 minutes. Tho careful doing that because the ISP might not like due to so many tiny packets making there network switches catch fire and later on getting a few calls from people with supposedly "secret" IP addresses on where you got there IP and why you attempted to connect to it.

author: Richard Crowley
author_rank: Super Contributor
timestamp: May 14, 2019, 09:19:54 pm
content: One of my favorite quotes from computing history:QuoteRef:https://en.wikiquote.org/wiki/Andrew_S._Tanenbaum

author: Beamin
author_rank: Super Contributor
timestamp: May 27, 2019, 11:34:52 am
content: Quote from: tooki on May 09, 2019, 10:02:39 pmWay before 2008 I remember we were looking at peoples webcams and doing other shit. I miss the days of the mail bomber and the anon email program where you could email as anyone else you wanted to. But people were more wise on the internet back then.He was looking at 10 people using google and didn't stop to think, "hmmmm it's 2008, maybe there are MORE then 10 people using google at any given moment, and the days of only having 10 people per site was probably before I was even born." Isn't .rr.com the roadrunner service of timewarner cable? That video is sad for 2008 at his age. Don't forget to subscribe before he goes through puberty. What was he actually looking at?

author: tooki
author_rank: Super Contributor
timestamp: May 27, 2019, 05:09:09 pm
content: Because he included the “http://” in the traceroute, he was trying to resolve a hostname that doesn’t exist. But because the ISP uses one of those services where failed DNS lookups don’t fail, but instead return a search page, traceroute was actually tracing the route to the ISP’s search page...

author: Beamin
author_rank: Super Contributor
timestamp: May 27, 2019, 08:16:39 pm
content: Quote from: tooki on May 27, 2019, 05:09:09 pmSo he did have roadrunner internet. He should have noticed this, but then again he sucks at "the intrawebz". I should write a manual on how to properly look at things to understand what's going on, especially since I'm legally blind and even I saw that.Just watched that video again and he couldn't figure out "elps" was el paso texas. That's really cringe. He can't view their internet because its better then his, I like that part. He sucks at BS'ing as much as he does hacking.