OT: Backing up video files long term - Page 1
Forum section: electronics / general

author: EEVblog
author_rank: Unknown
timestamp: July 03, 2014, 03:21:33 am
content: Opinions invited:What is the best way to physically backup all my raw (and edited) video files for the long term?I can fit maybe 500 episodes + other misc and family stuff onto a 2TB drive, and I currently have a NetGear ReadyNAS RAID system to do that, plus the odd drive here and there, but it's close to full.Yes, I know that RAID is NOT proper "backup" so please spare me the lectureI'm thinking maybe 3 separate hard drives in 3 locations per every 500 episodes?Or should I keep multiple RAID disk arrays, enclosures and all in different locations?Online cloudy solutions will likely eat into my data allowance at the lab, need to look into that, but can't rely on it solely.

author: free_electron
author_rank: Super Contributor
timestamp: July 03, 2014, 03:37:12 am
content: get a few harddisk in usb enclosure.for long term storage : two sets. one off-site ( at home ) one in lab. preventing fire disasters.drives are to be in a good schock munted enclosure and powered off. get drives that physically retract heads off-platter.drives are to be filled with data in 1 shot. then powered down until needed. let's say you got 2.3 terabyte of data : 'seal' 2 terabyte onto two 2 terabyte drives.the remainder is kept alive on your nas system (until you have collected enough to fill another drive.i would use 1 terabyte drives. that way you can fill the drive faster and 'seal it' ( the goal is to mimimize the amount of power cyles on the drive. )we d this for project data. once a project is over an the design can be archived it is dumped to a set of brand-spanking new harddisks in one shot. they are written , verified and then powered off.one drive is stored in the archival safe on site , one drive goes to an archival company. third and fourth sets are created at other sites around the world through our network. even if a drive is not full it is considered 'spent' on a project. drives are cheap.avoid nas boxes as the plague for this.. nas boxes often use a custom or weird file system. if the nas hardware dies the drive is unreadable unless you have an identical nas. never mind the nas may use ext3 or anther linux filesystem. if you have it formatted in a linksys nas it will not be readable by a wd nas... even though they both use linux internally. there is metadata and journalling and other proprietary stuff and that may not be compatible between nas boxes from different brands !a drive in usb enclosure does not have this problem. they are simple sata drives formatted in whatever filesystem used by the computer that wrote them. if the usb translator dies : plonk another one on there and it will work. worst case hook it to an esata or sata port and it will work.

author: XOIIO
author_rank: Super Contributor
timestamp: July 03, 2014, 04:25:24 am
content: I can't recall for sure, but don't tape drives have pretty decent storage if kept safe (eg away from magnetic interference sources)?I'd say just some standard drives in some insanely good packaging would be the best option though.You could also laser cut the binary into metal sheets to decode later

author: EEVblog
author_rank: Unknown
timestamp: July 03, 2014, 04:27:42 am
content: Quote from: free_electron on July 03, 2014, 03:37:12 amDon't they all do that these days?I've got a 3.5" hot swap sata dock in my lab machine, so can use that to backup onto individual drives and then store them in different locations.If I used 2 drives in two locations, plus maybe a cloud solution as well, then that's gotta cover all eventualities.I'll keep the NAS for local use on the most recent stuff.My desktop editing machine has two hard drives (in addition to the SSD boot drive), incrementally mirrored daily.

author: KK
author_rank: Regular Contributor
timestamp: July 03, 2014, 04:28:59 am
content: I use LTO tapes. I've been using tape for decades quantum-DDS-DLT-LTO.Copies on site, and in a bank vault.LTO-4 drives use 800/1600GB tapes. Video will be somewhere between the two numbers depending on compressibility.LTO-6 drives are out now with 2.5/5.0TB capacities.Good long term archival storage. The LTO-4 drive does 550GB/hr using A SCSI interface.LTO-6 is up to 1.4TB/hrI have had situations where I needed a backup copy off 8 year old tapes. And it was not a problem for several TB's of restores.I think optical storage has around 20 years or so shelf life because of eventual rot, and the magnetic tapes are around that lifespan.

author: EEVblog
author_rank: Unknown
timestamp: July 03, 2014, 04:31:41 am
content: Currently backing up 61 episodes worth of footage (284GB) to the networked NAS, and that's going to take 8 hours at 10MB/sec

author: SeanB
author_rank: Super Contributor
timestamp: July 03, 2014, 04:42:10 am
content: Careful on tape. Very long life, but if the drives are no longer made they are essentially useless ornaments. I have old backups made on assorted tapes, that are no longer usable as the drive or the software no longer is usable on modern computers.You might look into a BD recorder and store the video as data on them, making 2 copies as additional insurance. Store the discs in jewel cases inside a brown paper envelope, and keep in a box in a cabinet. A lot more space, but there you at least can read them with current drives, and the drives likely will be around for a while. No proprietary software needed to read them.

author: XOIIO
author_rank: Super Contributor
timestamp: July 03, 2014, 04:45:48 am
content: Good to have confirmation that tape drives do work well as I thought, I have a dell powervault tape storage system, it's pretty sweet, I just need the cable to hook it up to my server.Quote from: EEVblog on July 03, 2014, 04:31:41 amWould it not be faster to copy it to an external drive then directly onto the NAS, rather than over the network?You could also announce your channel is closing, when that happened with photonicinduction there was a torrent of every single video he uploaded, it's probably still alive, I should re-download it to watch some of his old stuff.

author: Richard Crowley
author_rank: Super Contributor
timestamp: July 03, 2014, 05:02:08 am
content: I don't trust ANY kind of field-burnable optical media. Not LONG term archival storage IMHO, IME.99.99% of this planets most valuable data is backed up and archived (two different functions) on digital magnetic tape.While the concern for long-term viability of the format/equipment is a concern, reviewing (and re-writing, if necessary) archive tapes every n years is only prudent if the data is worth it.Hard drives are iffy, IMHO. I have too many of them that just died on the shelf in long-term archive service.If I wanted to do that, I would make at least two copies on DIFFERENT BRAND hard drives, and store them sealed in plastic zipper bags in separate locations (one at the office, and the other at home, etc.) The reason for different brands is critical because there have been several cases of bad batches of hard drives which created catastrophic problems for people "putting all their bits in one basket".I would also at least plug in the hard drives every year or two just to confirm that still spin up and are readable, etc.I use raw drives for archive storage and use a docking station like this to access them...http://www.amazon.com/Plugable-Lay-Flat-Docking-Station-ASM1053E/dp/B00APP6694And there are many storage options to protect raw drives when you aren't accessing them. For example...http://www.ebay.com/itm/like/300540328686?lpid=82

author: XOIIO
author_rank: Super Contributor
timestamp: July 03, 2014, 05:13:09 am
content: A really slick option would be to take the drive(s), find a good solid enclosure, cap off the top, fill the outside with spray foam all the way to the end, then have some other type of shock absorbing foam around that. Then maybe rubber bumpers and a metal case around that to close it off. You could also make a cover with spray foam if the metal case closed up.

author: TheBorg
author_rank: Frequent Contributor
timestamp: July 03, 2014, 05:18:45 am
content: Quote from: Richard Crowley on July 03, 2014, 05:02:08 amWould ESD be an issue? Seems to me a plastic bag would have the potential to fry circuits on the control boards. Agree on the different brands though.That being said, I store a couple of raw drives in a plastic drawer around here...

author: EEVblog
author_rank: Unknown
timestamp: July 03, 2014, 06:04:42 am
content: Quote from: XOIIO on July 03, 2014, 05:13:09 amPelican case!

author: XOIIO
author_rank: Super Contributor
timestamp: July 03, 2014, 06:36:35 am
content: Quote from: EEVblog on July 03, 2014, 06:04:42 amPretty much, I think the spray insulation foam might be a good other layer to add, then put it in a pelican case that just pinches it between the foam cones.Although I think we are getting into territory of dropping the hard drives, from considerable heights rather than just storageGuess it would be good to have videos survive a nuclear blast though

author: abaxas
author_rank: Regular Contributor
timestamp: July 03, 2014, 09:00:11 am
content: This doesn't sound like a backup issue at all. It's an archiving one.The first question should be, do you actually need to keep the raw video? I know it's human nature to hoard everything but does it actually have any use?

author: EEVblog
author_rank: Unknown
timestamp: July 03, 2014, 09:44:09 am
content: Quote from: abaxas on July 03, 2014, 09:00:11 amYes, basically archiving.QuoteI want to keep it for the reasons Ican'tthink of right now!One basic requirement is that the original source material is higher quality than what is uploaded to youtube.Recent ones are pretty close in quality, but anything below say 400 or so was optimised for upload time and not video quality.And there are potentially outtakes and other extra stuff.Plus the editor project files, and I can go back in and re-render the original video if needed, or make new variations, extended versions etc.It's silly to just throw it all away if I can at the very least store it all on a few hard drives that cost sub $100 a pop. So might as well do that.Lets say it costs me a few hundred dollars every 500 videos in storage, that's pretty cheap I think.

author: EEVblog
author_rank: Unknown
timestamp: July 03, 2014, 10:29:06 am
content: Yes, I'm considering getting a quality Bluray drive and brand name discs as an extra backup.Can probably fit 5 average episodes on each disc.Seems you can buy x8 and x16 write speed BR drives, but only x4 write media?

author: Legit-Design
author_rank: Frequent Contributor
timestamp: July 03, 2014, 10:52:15 am
content: Overspeed writing? Like in normal CD-R times, could write a cd really fast. Or could write it really slow and be sure it would not crap out at some point, when some bit didn't settle there correctly.http://www.soundonsound.com/sos/nov04/articles/qa1104-3.htmQuoteSo there might be some difference in terms of reliability. I'm sure some audiophiles can also hear the difference?

author: XOIIO
author_rank: Super Contributor
timestamp: July 03, 2014, 10:54:37 am
content: Never tried burning standard files with imgburn but it should allow you to verify those as well, it has that option for ISO's at least, it helps make sure there were no mistakes.

author: Legit-Design
author_rank: Frequent Contributor
timestamp: July 03, 2014, 10:56:23 am
content: Quote from: XOIIO on July 03, 2014, 10:54:37 ammd5sumsha1sumworks every time.

author: XOIIO
author_rank: Super Contributor
timestamp: July 03, 2014, 11:07:23 am
content: Quote from: Legit-Design on July 03, 2014, 10:56:23 amWhat is the difference between the two? Never heard of sha1sum, a bit of googling just hints that it's a bigger hash? Also saw mention of SHA256

author: Legit-Design
author_rank: Frequent Contributor
timestamp: July 03, 2014, 11:20:25 am
content: Quote from: XOIIO on July 03, 2014, 11:07:23 amMD5 collisions. Security. Tinfoil hats.Remember stuxnet? And it's close relative Flame?http://arstechnica.com/security/2012/06/flame-crypto-breakthrough/Some really big entity took really smart people and state of the art supercomputers, then they made something that was not suppose to happen and made it happen.

author: SirNick
author_rank: Frequent Contributor
timestamp: July 03, 2014, 06:47:29 pm
content: Disc chemistry will dictate the best burn rate. Or at least it has for previous optical media formats. Don't over-speed them, don't under-speed them. Choose a speed that it seems to like best and stick to it. I just let the drive choose unless I have some reason to override this. I have CD-Rs I burnt back in circa 2000 that are still readable. Good burner hardware and quality media is the key here.So, what I would do in your case is this: Buy yourself the largest established HDD at the time. That is, don't buy a 4TB when they're new. Stick to the 2TB models for a while. This is how I populate my (home-built rack-mounted Linux MD RAID) NAS. When the 1TB drives filled up, I took them out and replaced them with 2TB drives. I then used the 1TB drives to back up the NAS. Since I wasn't losing 1/4 of my space due to RAID-5 overhead, this was pretty much enough until I filled up the 2TB drives. I'm about due to replace those with larger disks, and the cycle begins again. This way, you're not leaving drives for decades, but constantly refreshing the hardware. Just use a USB dock to copy from NAS to backup HDD, then store it somewhere safe. You don't need to bomb-proof the things, just put them somewhere they won't be disturbed, in the little plastic clam-shell the new drives came in.It would also be wise to back up individual episodes (or a couple, depending on how much space it takes) on BD-R. Keep these in sleeves in a binder or storage box so you can go right to the material of interest if/when you decide to revisit old footage. If the disc fails, go to your HDD backup. This will keep the power-cycle count low on the spindle motor. The optical disc won't care how much it's used.MD5Sums of all the data couldn't hurt, so you can tell if there was bit-rot along the way. But usually, the medium itself will have a CRC of its own at some level that will fail in obvious ways (read errors) so, meh.. Keep in mind, it's merely an alert that something has gone bad, not prevention, so consider the worth of that feature before going to any trouble. I do have a script that updates a per-directory md5sum on my media share for this purpose, so I can eventually write another script that compares this to the actual file checksums and lets me know if a movie file has gone bad.

author: nctnico
author_rank: Super Contributor
timestamp: July 03, 2014, 07:21:14 pm
content: The problem with storing hard drives, tapes, disks, etc is that you don't know whether it is good or not until you plug it in. Interfaces and media becoming obsolete is also a problem. Nowadays I keep backups on 3 seperate disks in 3 seperate systems which are synchronised using rsync every night and supplemented with the work I did during the day. Because the drives are active I know immediately when one of the drives has failed so I can take immediate action to replace it. If the disks are full I just add/replace the disks. It acts more or less a distributed RAID array which protects me from losing data by fire, flooding and theft.

author: pickle9000
author_rank: Super Contributor
timestamp: July 03, 2014, 07:36:02 pm
content: I have used every kind of media over the years for backup but I find now I tend to use two hard drives, fill, unpowered, and store one off site. I use exactly the same drive and I prefer WD drives (personal preference).Two years ago I transferred many of the files from old drives / media to a pair of new hard drives. I did actually find the odd file fault but was able to recover a good one from the other source. So for me the lesson is two backups are far superior than one regardless of the media.The only other thing is that I keeps rough paper records with the drives. That has proven to be a very big plus over the years.

author: SirNick
author_rank: Frequent Contributor
timestamp: July 03, 2014, 10:59:52 pm
content: Quote from: nctnico on July 03, 2014, 07:21:14 pmStill.... RAID and rsync offer zero protection from faults above the physical layer, like file system corruption or accidental deletion. All it takes is one virus or a rogue rm -rf and everything's gone. As long as you catch this before it cascades, great. You're working within a window constrained by single-point-of-failure on one end, and margin-for-error on the other.Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up. (I'm sure it's happened in the history of computing.) Ihavehad some develop media errors over time (so have two backups if the data is important), but the usual fate is throwing away those that were just too small to be of any practical use anymore.

author: EEVblog
author_rank: Unknown
timestamp: July 03, 2014, 11:07:01 pm
content: Quote from: SirNick on July 03, 2014, 06:47:29 pmYes, I think I'll do this in addition to the two hard drives.

author: EEVblog
author_rank: Unknown
timestamp: July 03, 2014, 11:08:30 pm
content: Quote from: pickle9000 on July 03, 2014, 07:36:02 pmAs someone else noted, I wouldn't use the exact same drive for both backups, but two different brands.I think I'll go buy a bunch of 1TB WD and 1TB Seagate drives.

author: EEVblog
author_rank: Unknown
timestamp: July 03, 2014, 11:10:18 pm
content: Quote from: SirNick on July 03, 2014, 10:59:52 pmAny archive or backup system, regardless of how complex, is a simple (or maybe not so simple) matter of probability theory.

author: ovnr
author_rank: Frequent Contributor
timestamp: July 03, 2014, 11:47:47 pm
content: Well, if you want an online solution, Amazon's S3 Glacier storage is pretty nice. $10/month per terabyte stored, pretty reliable ("99.999999999% durability of objects in storage", survives the complete loss of two datacenters). Obviously more expensive than disks over time, especially for massive heaps of data - but I consider it manageable. I don't use it myself (yet) - I currently run two 6x2TB RAIDZ-2 arrays and generally swap disks when they get five years old; this costs me around $400/year compared to $2000/year, but it's offsite and as reliable as you can hope for.I'm going to be replicating all of my irreplaceable data onto S3 eventually; it's just a couple TB or so. The rest can be rebuilt, but at considerable cost.

author: pickle9000
author_rank: Super Contributor
timestamp: July 04, 2014, 12:23:26 am
content: Moving to a more modern (backup) system allowed me to reduce the physical size of the backups, and throw out some very old (large and cluncky / slow) hardware. I'm not saying it was unreliable just old. It was also good to see that the data was still viable.The main motivation was to reduce the size from that of a couch sized space to that which could fit in a shoebox. Remember not just the media and or drives but a couple old computer systems to go with it.

author: EEVblog
author_rank: Unknown
timestamp: July 04, 2014, 12:24:52 am
content: Quote from: wilfred on July 04, 2014, 12:14:06 amBecause the point is that if you use two identical drives then you have the added (albeit small) percentage that a design or manufacturing defect in that model of drive will render both of your backups useless in X years time.Especially so if you buy two identical models from the same manufacturing batch. Likely if you go to the same store to buy them, as they will likely get them in boxes from the same batch."The most reliable" model is data that is essentiallyonlyavailable in hindsight.

author: Monkeh
author_rank: Super Contributor
timestamp: July 04, 2014, 12:36:46 am
content: Quote from: EEVblog on July 03, 2014, 04:31:41 amAnd that's why they invented gigabit ethernet. 15 years ago.

author: pickle9000
author_rank: Super Contributor
timestamp: July 04, 2014, 12:45:16 am
content: Quote from: EEVblog on July 04, 2014, 12:24:52 amI agree that using 2 different brands is a better system than what I have, no question. I do run the WD drives I use for a few hundred hours to help ensure they are up to snuff. That's not the same as using completely different hardware. My personal preference for WD relates to problems with the other brand Dave mentioned. Even so I will probably throw one into the mix next time I do a backup.

author: Richard Crowley
author_rank: Super Contributor
timestamp: July 04, 2014, 01:17:30 am
content: Quote from: SirNick on July 03, 2014, 10:59:52 pmI have seen SEVERAL (six or eight) hard drives in long-term archival service simply be DOA when you try to power them back up. This is why a single hard drive is NOT reliable backup or archival media IME. I have also seen raid arrays fail because they used several drives from a single defective batch.Hard drives are OK for backup and archival storage, as long as you take appropriate precautions.

author: David Hess
author_rank: Super Contributor
timestamp: July 04, 2014, 02:17:22 pm
content: Quote from: mojo-chan on July 03, 2014, 10:17:00 amBluRay is not even close to magneto-optical except maybe in reliability and I have doubts about that. I still use my 3.5" magneto-optical drives which hold up to 640MB.I would consider archival BluRay a good choice except for price. Even multiply redundant hard drive storage is less expensive but I wonder what their unpowered lifetime is.

author: David Hess
author_rank: Super Contributor
timestamp: July 04, 2014, 02:32:38 pm
content: I have had good results backing up onto good single layer DVD+R disks but those will be too small for your use. I generate and save PAR2 recovery files along with the data and write the disks out at the slowest burner speed with verification and both UDF and ISO file system data so in the event of a damaged disk, recovery is easier. The only disks failures I have had in 10 years were either do to cheap media or actual physical damage and almost all of them were completely recoverable. I suspect BluRay would be similarly reliable under these conditions.For larger data sets I would go with two identical file servers running Linux or BSD using 4 disk RAID5 or perhaps one of the newer file systems which support data integrity like ZFS or Btrfs.

author: nctnico
author_rank: Super Contributor
timestamp: July 04, 2014, 03:46:07 pm
content: Quote from: Richard Crowley on July 04, 2014, 01:17:30 amI disagree. That particular failure mode is (often) caused by hard drives running too hot. If you keep hard drives cool you won't have this problem.

author: mariush
author_rank: Super Contributor
timestamp: July 04, 2014, 04:19:08 pm
content: I thought about this as well and came to the conclusion that just plain hard drives in a RAID is the most future proof in long term.Bluray is only ok for temporary storage, a few months to a year. After that, just like with DVDs, you can't really trust the media. It's cheap enough (2-3$ for 25 GB) to have a 3rd backup on blurays but not as a primary backup.Tapes are cool, lots of storage relatively cheap and in small space but like others said, you have the problem of continuously having to upgrade your tapes to whatever comes up next year or to keep buying tape reader/writers (and have some as backups) just in case they're discontinued at some point and your main unit breaks down.Even though I respect free_electron and his experience, I don't agree with his suggestion of turning on hard drive, storing the data and then turning off the drive. Maybe it's silly, but I don't trust a hard drive NOT failing at startup all of the sudden, I'd be more comfortable with having the hard drive running 24/7 and potentially catching some error in close to real time and have the hard drive replaced.I would go with one of those units that allow you to connect 4 or more hard drives and set up a raid5 or better and check from time to time the state of the disks and take corrective measures when needed.In your case, you could have an identical unit at home (but maybe with other drives or another series just in case there's a bad batch) and you could just configure the home unit to slowly copy throughout the day what you have stored at the office (doesn't have to fast, it can be just 1mbps upload for example, you have it running 24/7 anyway)Maxxarcade (of youtube "fame") recently upgraded his raid storage from a server with lots of ATA drives to a Synology DS1813+ (http://www.newegg.com/Product/Product.aspx?Item=N82E16822108138) that can hold 8 hard drives, has 4 network ports that can be linked together for higher throughput etc etc. It's a bit expensive, but there's the cheaperDS1513+also which can only hold 5 drives.I think it's a good investment if you're serious about backup.PS. For extra security, you could use a tool like PAR2 to create recovery volumes for the video files. With about 20% of the video size in recovery volumes, you can recover pretty much any video file if it's just a bunch of bits or bytes messed up randomly (it won't help if there's a continuous 10-20 MB of totally damaged data)md5sum and sha1sum will just make a checksum and that tells you if the file got corrupted or not, but par also is capable of repairing the files.

author: rob77
author_rank: Super Contributor
timestamp: July 04, 2014, 04:30:01 pm
content: ok i didn't read the whole thread, but my 2 cents :1. blue-ray - do you guys have experience with 5+ years of storage of those ? i guess not. if blue-ray (it's cheap) then refresh every 2-3 years to fresh media.2. tape drives - most expensive but it's a industry standard and very reliable. the best for archiving are the WORM LTO tapes (Write Once Read Many) archived in controlled environment (humidity, temperature). but as mentioned it's freaking expensive solution to set up, with a good price per TB once it's there, but still i wouldn't recommend this for archiving videos just because of the price tag (not just the hardware and tapes, but the archiving of the tapes in a controlled environment gets expensive). and let's not forget - you would need to refresh the tapes as well - it's usually 10+ years guaranteed for properly stored WORM media.3. hard drives - cheap and reliable to some extent. but still i would recommend to keep 3 independent copies offline and regular check/refreshi think at the end of the day it doesn't matter whether blue-ray or HDD - the cost is very similar for those methods and both would require regular refresh.

author: madires
author_rank: Super Contributor
timestamp: July 04, 2014, 04:40:48 pm
content: Quote from: free_electron on July 03, 2014, 03:37:12 amThe problem is based on the version of the linux kernel and the RAID tools used. I've managed to rescue NAS disks (RAID 1) by attaching them directly to a recent linux system. But I agree that USB disks are the best choice for backup at the moment, unless you have to backup a datacenter.

author: David Hess
author_rank: Super Contributor
timestamp: July 04, 2014, 05:44:55 pm
content: Quote from: mojo-chan on July 04, 2014, 03:36:52 pmThe coercivity of the media in hard drives is incredibly high to support their high data density in the face of thermally activated relaxation so external magnetic fields, even strong ones, have an insignificant effect. Magneto-optical media is tough for the same reason.QuoteI agree about using them for long term offline storage although I suspect they will fair better than many alternatives. For that reason, I would use them in a small self contained server and run them through a complete background scrub while updating the contents.QuoteAt least where I am in the US, internet based storage is totally useless because of both bandwidth and ISP transfer caps.

author: IanJ
author_rank: Unknown
timestamp: July 04, 2014, 08:37:57 pm
content: Hi,My own backup solution for my Vegas Movie Studio video files is a 2nd drive on my PC where the files sit, and a QNAP NAS running RAID 5 (4 drives).I also have a 2nd QNAP NAS running RAID 1 (2 drives) which is where I backup all my other data etc including my 1&1 dedicated server (web & email) backups.All drives on both QNAPs are WD Enterprise edition RE3 drives.Works for me, dead easy to administer. Have only ever had 1 drive failure (a 13 month old WD RE3) on the RAID 1 NAS.Tips: Keep the server cupboard temperature stable, keep it dust free, and install a wee UPS.Ian.

author: XOIIO
author_rank: Super Contributor
timestamp: July 04, 2014, 11:39:59 pm
content: Quote from: IanJ on July 04, 2014, 08:37:57 pmI'll guessing you use one that sends a shutdown signal when the batteries get below a certain percentage in case of long term power failure?

author: Rigby
author_rank: Super Contributor
timestamp: July 05, 2014, 12:14:35 am
content: The Library of Congress, when they designate something worthy of the time and effort required, save copies on hard disks and verify the data periodically, then when new hard disk tech comes out, they copy to new media while drives for old media still exist and are serviceable.Last I heard they use a mix of LTO tape and hard drives.LTO isn't going anywhere, and the new drives that come out are backwards compatible, IIRC. LTO is pretty safe. Hard disk is pretty safe. I wouldn't bother with optical anything.

author: SL4P
author_rank: Super Contributor
timestamp: July 05, 2014, 04:52:28 am
content: As mentioned previously, you are archiving.You need to look a the access requirements vs the cost / time to retrieve a specific title etc.Read about HSM strategy (Hierarchical Storage Management), and consider the various combinations of online, near-line and offline repositories. The use of stub files can also provide near-instant online access, while loading the large part of for complete playout while the stub is playing.

author: IanJ
author_rank: Unknown
timestamp: July 05, 2014, 09:52:36 am
content: Quote from: XOIIO on July 04, 2014, 11:39:59 pmThat would be a bonus but actually my own UPS I fitted to help with a few 15 minute outages I have had in the past couple of years. The batteries have always lasted.I also have the UPS keep my modem/routers alive.One other issue with NAS boxes is I think it is important to power cycle every 6 months. I had a raid drive fail on one of my qnap NAS boxes, it was fine right up to the power up when the drive just failed to start. Hidden problems! Can't imagine maybe more than 1 drive being affected and then suddenly after a power cycle no data!Seriously, for what Dave needs I would just stick with a good NAS box with quality drives.......and use it for other data also.Ian.

author: free_electron
author_rank: Super Contributor
timestamp: July 05, 2014, 10:14:25 am
content: Quote from: mojo-chan on July 04, 2014, 03:36:52 pmI'm gonna be very blunt here : Stay off of whatever you have been smoking because you are talking out your ass. What you are saying is pure nonsense.-The surface of the disk is not the recording layer ! That layer is buried. The surface layer is a lubricant with underneath a layer of cristalline carbon (basically grown diamond)Then there are several other layers before you even hit the actual recording layer.-earth magnetic field have no impact. You can put a permanent magnet on a harddisk case while it is running. No harm will happen to the data.-all platters have bad zones. These are NOT mapped on the fly. The mapping happens during manufacturing and is stored in large lookup tables on the drive. If, during operation, the drive encounters problems with a sector it can add this to the existing table. The surface is 100% tested during manufacturing when the drive formatter writes the servo signals and sector marker wedges. This is information is doublechecked and any problems are marked in the lookup table. All harddisks go through a 24 to 48 hour burn-in cycle. It takes about 3 hours for the formatter to do its work ( do not confuse the formatting process in the factory with formatting a filesystem. It has nothing to do with each other. Once a drive is completely assembled and sealed you cannot alter the physical formatting as you need an additional head that is now no longer there. If you look at a harddisk you will find silver metallic small labels , either on the side , or between the cricuit board and case. They cover a hole and or slit. That is where the formatters head enters the drive to write the synchro track on the platters.-as for the rewriting of data. That is absolute nonsense. We try to minimise the amount of writing.I design those things for a living. None of what you are saying is true.

author: free_electron
author_rank: Super Contributor
timestamp: July 05, 2014, 10:19:06 am
content: Quote from: madires on July 04, 2014, 04:40:48 pmThe problem is each nas seems to use its own thing. There are rarely updates to the firmware , some have bugs, some use reiser, so e use ext 2 , so e ext3 , some have journaling on, others off.Nothing but trouble. Some even store the runtime image on the drive.Usb box, format drive in ntfs and off you go. All computers can read ntfs these days. No need to screw around with obscure things .

author: Richard Crowley
author_rank: Super Contributor
timestamp: July 05, 2014, 10:20:46 am
content: Quote from: free_electron on July 05, 2014, 10:14:25 amYou don't even have to put a magnet on your hard drive. If you disassemble one, you will end up with one of the strongest permanent magnets you have in the house. So strong, you could seriously hurt yourself with one if you got pinched between it and a piece of ferrous metal.You can find all sorts of complete twaddle on the interweb.caveat lector

author: Richard Crowley
author_rank: Super Contributor
timestamp: July 05, 2014, 10:26:27 am
content: Quote from: mojo-chan on July 04, 2014, 03:36:52 pmSome of what you say is true of "non-volatile" solid-state memory (i.e. "flash memory"). Which is why nobody I know who understands the physics considers NVram to be suitable for "archival" storage.But NONE of what you wrote has anything to do with conventional rotating-disk hard drives. Complete nonsense.

author: magetoo
author_rank: Frequent Contributor
timestamp: July 05, 2014, 10:45:26 am
content: Quote from: free_electron on July 05, 2014, 10:14:25 amDo you have a reference you could point to?Asking because I've been hearing the exact same things that mojo-chan is saying: hard drives nowadays work not because they are error free, but because error correction is so good that the drive can cope with the inevitable faults; that those faults are a part of normal operation; that reading through a drive's sectors is necessary to spot "bits that are getting weak" (hand waving furiously) so that the controller can do its remapping magic, etc.Would be nice to have something more to go on than "well, a guy on a forum said".

author: magetoo
author_rank: Frequent Contributor
timestamp: July 05, 2014, 11:15:35 am
content: For published episodes, maybe archive.org could be an option? I know many people seem to be using them for video podcasts.Since you have a ton of them, I would think that there might be some way to solve the problem of getting the bits there other than uploading (like mailing drives). Their FAQ says to contact them, but doesn't go into details.

author: EEVblog
author_rank: Unknown
timestamp: July 05, 2014, 11:22:30 am
content: Quote from: magetoo on July 05, 2014, 11:15:35 amNever looked into it.Although isn't that public only?I consider my raw video files private, and not something I'd generally publicly release as a whole.

author: magetoo
author_rank: Frequent Contributor
timestamp: July 05, 2014, 11:48:08 am
content: Yes, it's not an option for everything. But for things that are public anyway, the EEVblog episodes, it would be free backup.(And we viewers get another place to get them from.)Edit: But maybe I've misunderstood what you mean. I guess the finished product is small compared to the raw video, so maybe that's not so relevant.

author: EEVblog
author_rank: Unknown
timestamp: July 05, 2014, 12:45:44 pm
content: Quote from: magetoo on July 05, 2014, 11:48:08 amYes, finished and uploaded video is maybe 1/5 the size of the original raw files.

author: SeanB
author_rank: Super Contributor
timestamp: July 05, 2014, 02:27:53 pm
content: Quote from: magetoo on July 05, 2014, 10:45:26 amYes, it is good to refresh the drive info by doing a read every so often. SMART can do this if you ask for a full surface scan, and will take a few hours to run this, and will read all the data on the drive and see if it is all readable after error correction. If the error bursts get too big ( drive firmware dependant) for a sector the drive will then rewrite the corrected data and try again. If it decided the sector is no longer reliable it will use one of the spare sectors it has and add the bad sector to the bad block table and write the data elsewhere.You can use SMART to do this or use other tools, the most advertised is Spinrite from GRC.com which can recover most data from bad blocks and try to get a last good read from failing bits by statistical analysis of bad bits in marginal cases. It also will do the same for failing MLC and SLC drives where the data slicer gives a bad reading.If you ever have used Norton Disk Doctor to recover data off a flexible disk, or used TIP to recover a ZIP disk ( I have used it a lot and it works well on ZIP drives with poor sectors to recover the data) you have used Steve Gibson's software invention.

author: SL4P
author_rank: Super Contributor
timestamp: July 05, 2014, 02:41:05 pm
content: Someone should mention that any dynamically writeable media (disk / tape / USB etc), that is visible as a logical volume - is still fair game for malware attacks.The only 'archive' is not mounted, and the only 'safe archive' is rotated through offsite repositories durong the refresh/resync cycle.

author: madires
author_rank: Super Contributor
timestamp: July 05, 2014, 02:53:16 pm
content: Quote from: free_electron on July 05, 2014, 10:19:06 amActually, the various filesystems aren't a big problem. If the mount command can't detect the filesystem automatically you simply try all possible filesystems (read-only of course). And for the linux extended filesystems you got several choices to get your data. ext3 is ext2 plus journaling, i.e. you can mount an ext3 fs as ext2 read-only and read the files. It's also possible to recover a linux based RAID 5 if the disks are still ok. But that's not a task for the average user.QuoteLinux users prefer ext4

author: nctnico
author_rank: Super Contributor
timestamp: July 05, 2014, 03:23:26 pm
content: Quote from: mojo-chan on July 05, 2014, 02:18:33 pmIf that is your opinion you have not been reading too well. The consensus it to keep a backup on multiple hard drives with 24/7 health monitoring. If one fails you replace it and you gradually upgrade to new technology. A bluray disc isn't redundant and if a new technology becomes availabe you will have issues finding a backwards compatible reader.

author: David Hess
author_rank: Super Contributor
timestamp: July 05, 2014, 04:22:10 pm
content: Quote from: mojo-chan on July 05, 2014, 02:18:33 pmBluRay is not magneto-optical; it does not use any magnetic layer, does not use the Kerr effect for reading, and has no bias electromagnet. It uses a spin-coated dye or pigment or a sputtered alloy or composite just like a recordable or rewritable DVD.

author: free_electron
author_rank: Super Contributor
timestamp: July 05, 2014, 05:06:18 pm
content: Quote from: Richard Crowley on July 05, 2014, 10:20:46 amThe field of that magnet is trapped in the horseshoe around the vcm coil.. But you are right. Those magnets are the strongest there are. In fact, they are so strong that , if you place two of them near each other , they will 'jump' towards each other and, on impact, will shatter. The speed and force of impact is such that the magnets crack and chips and slivers fly off. This is very dangerous. Always wear safety goggles when dealing with these suckers.

author: Monkeh
author_rank: Super Contributor
timestamp: July 05, 2014, 05:32:42 pm
content: Quote from: SeanB on July 05, 2014, 02:27:53 pmSpinRite is an archaic joke and Gibson is a prolific liar.

author: free_electron
author_rank: Super Contributor
timestamp: July 05, 2014, 05:33:11 pm
content: Quote from: SeanB on July 05, 2014, 02:27:53 pmWHAT ?There is no such thing as a data refresh. There is no point in even reading periodically.The safest data is data stored on a harddisk that has its heads retracted off platter and powered off.-banging on it doesnt cause mechanical damage due to headstrikes as the heads are off disk , suspended and locked-viruses cant corrupt data as it is powered down-power surges can't fry the electronics as it is disconnected.-the drive is a dust free sealed environment (not like a plastic wobbly disk , exposed to degradation due to light exposure, easily scratched and full of greasy fingerprints)There is no need for surface scanning. The drives do this when reading as part of the reading process, but only for the sectors requested by the read. It is not true that they willy-nilly access passing sectors if they have no business there. A drive access only sectors requested by the host. If it encounters a read error or seek error it will retry a number of times and then mark the sector as bad. A bad sector will not prevent subsequent read attempts. It will prevent write attempts.There is also a big difference between logical and physical mapping. You have no idea where your data really resides on the drive. Your data is also not stored as the chain of ones and zeroes you feed the disk. The data has redundancy added to it and the packet then goes through a scrambler to prevent having more than 5 consecutive ones or zeroes. One byte you feed is stored as 10 bit on the media.Throw in the servo wedges and other stuff like lookup tables and you end up that a drive holding 1 terabyte of user data, actually stores about 1.3 terabyte. Your data is stored as 'symbols'. 5 bits are used to store 4. Depending on the order of symbols we can flick the symbol polarity.Lets say we find out that two consecutive symbols end up being 01000 and 00010. This would give six consecutive seroes on the drive. We replace the second symbol with 11101 as this has the same user payload. A symbol can be stored as real or as inverted. It doesnt matter symbols are symmetrical. Additional symbols, containing redundant information are injected in the stream.Additional techniques like viterbi encoding and maximum likelyhood algorithms can weed out even multi bit errors. The mathematics below it is very complex.Optical disks use a similar mechanism. They actually store 11 bits for every byte you feed them... The 8 bit is expanded to 14 and then compressed to 11(overall compression reduces the 14 bit streams to an average of 11. Some packets are less compressable than others. This compression algorithm is tuned for music on cd's and video on DVD and bluray. The compressors also work on data of course but they are tuned for video or audio

author: rob77
author_rank: Super Contributor
timestamp: July 05, 2014, 05:46:48 pm
content: Quote from: free_electron on July 05, 2014, 05:33:11 pmrefresh to NEW media ! and yes it's a common practice. so refresh to NEW harddrives, NEW blueray media, NEW tapes.if you understood it as reading and writing back , then your understanding was wrong

author: free_electron
author_rank: Super Contributor
timestamp: July 05, 2014, 05:49:57 pm
content: Quote from: Monkeh on July 05, 2014, 05:32:42 pmSpinrite works fine on very old RLL drives (the ones that needed dedicated controller cards with two flatcables). It also worked on early ide drives. His idea was fine and worked. But any drive above 300megabyte uses other techniques internally You cannot acces raw head data like you could in old drives.Spi rite today is a good program that simp,y does multiple retries on bad sectors. Normally the drive will do a fixed number of tries , flag an error and give the datablock , errors included, to the host. And that's it. Spinrite is nothing more than a tool that, on a user level, keeps sending retry operations, grabbing the returned block , comparing them and filtering out what data is 'static' and what data is destroyed.A block can be a few kilobyte. A damaged block is not completely corrupt. It may just be a few bits in the stream, there where the mechanical damage is(and the magnetic layer is removed). Only those bits will change from retry to retry. The rest stay static. So yes, you can recover partial data by collecting two or three retires and finding the static bits. If you are lucky and the filesystem also uses some form of error correction and redundancy you may retrieve the data. If there is a catastrophical failure and entire sectors, including the serve wedge, are gone there is no recovery. If the servo wedge and track data block (not the user data but the data used by the drive to find a sector) is destroyed you cannot find it. It can be retrieved on a spin stand but the drive itself does not have the capability to do that. It would cost too much to give the drive such mechanical precision. We simply employ electronic servo mechanisms to keep the head on track. If there is no indicator where the track is, the servo doesnt work. The drive does not have the mechanical precision to put it on a track and leave it there. A spinstand can do that, a drive can't.I can go on for days explaining all the trickery and mechanisms involved. It is a very complex system, devised for maximum data safety.I trust drives much more than optical (self recorded) media. Take a recorded dvd and let it lay in you office exposed to ambient light. Many degrade after a year or so. A lot of cheap ones use an organic material that degrades in light. So even if you keep em in the dark : when reading them the laser in the drive is exposing them....There are long term storage grade disks havng a metal coating. But those can only be written at maximum 2 or 4 speed. I have very old recorded CD from traxdata. These are gold and you can clearly see the holes under a strong microscope. Recording those took 25 minutes for 650 megabyte. These disks are 16 or 17 years old and work fine. The last 'silver' one i wrote from emtec (basf) was corrupted after 3 months... Simply because it lay in the open.Gimme a nice cast aluminum frame as enclosure any time.But, keep three copies and store them in different places.

author: XOIIO
author_rank: Super Contributor
timestamp: July 05, 2014, 05:55:55 pm
content: Quote from: free_electron on July 05, 2014, 05:49:57 pmI'd love to see a picture of those.

author: David Hess
author_rank: Super Contributor
timestamp: July 05, 2014, 05:56:26 pm
content: Quote from: free_electron on July 05, 2014, 05:33:11 pmRAID controllers often include a scrubbing option which periodically reads an entire drive allowing the drive controller or the RAID controller to reallocate marginal or bad sectors. Sectors may be reallocated on read even when the data is good or at least recovered.QuoteThere is a logical to physical mapping table and at least on older drives, spare sectors are distributed over the entire surface. High end drives that used a dedicated servo track and supported low level formatting could optionally include a spare sector on every track lowing the number of reported bad sectors but also sacrificing capacity.

author: free_electron
author_rank: Super Contributor
timestamp: July 05, 2014, 05:58:01 pm
content: Quote from: rob77 on July 05, 2014, 05:46:48 pmThat's not what SeanB is talking about. Copying to new media is one kind of refresh. Fine with me.He was talki g about reading the same drive periodically as to scan the surface for damage.Data on a drive can o ly get damaged bqy mechanical impact (headstrike). So , by reasing you create an i stance where headstrike could happen. Are you sure that the data you just read is now still intact ? After all the head passes multiple times over the same track. The data marked ok in the first pass could now be destroyed during the second pass when accessing a different sector on that track. Oopsie.. Chicken and egg problem (and no, what you think is track 1 sector 1 and track 1 sector two are not necessarily adjacent to each other, in fact they may physicall lay on different tracks ! Physical and logical are two different things. You have no control over phyqsical, inly the drive knows. This is done te maximise throughput on the drive and shorten seek operations. I formation is staggered.

author: nctnico
author_rank: Super Contributor
timestamp: July 05, 2014, 06:08:15 pm
content: Quote from: free_electron on July 05, 2014, 05:33:11 pmBut you never know if it works until you power it up again. That is the big gamble besides other people grabbing the disk and use it for other purposes. And as others pointed out it may be impossible to replace the mechanical and electronical parts.

author: miguelvp
author_rank: Super Contributor
timestamp: July 05, 2014, 06:11:07 pm
content: Quote from: rob77 on July 05, 2014, 05:46:48 pmI agree, I've done many media refresh copies when we used tapes (the big ones like in the old movies) it was common practice to copy the media to new fresh tapes every so often (don't recall exactly how often now). Then send a copy off site in case of a fire or some other event that could destroy everything in site.Hierarchical backup was the craze later in the mid/late 90's not sure where they are at now but they where VERY expensive. They would have hard drives, digital tapes and optical media as well. with redundant power supplies and hot swappable RAID drives, the tapes and optical had may bays and some robotics where involved to move tapes and discs around.

author: free_electron
author_rank: Super Contributor
timestamp: July 05, 2014, 06:20:18 pm
content: Quote from: David Hess on July 05, 2014, 05:56:26 pmRaid comtrollers can scrub : but the drive firmware needs to suppport it. You will need SAS drives. Your runof the mill ide or sata can't do that.Dediated servo tracks no longer exist. That has gone away with the diskpacks. Damage to the servotrack was a catastrophy as the entire disk became unusable , true the head for servo was read only but a mechanical damage on that track was a disaster. In diskdrives as we know emfrom the PC world , servo data is interspersed with user data. That is the base operating principle of the drive.There is a servotrack created by the formatter to lay down the servo wedges. Once the wedges have been written that servotrack is no longer accessible. It has never been accessible by the drive.Here is how a drive is 'formatted' in the factory. You have a surface with no information on it. So how do you place markers ? The drive mechanics is not precise enough and only works if servo mechanisms can stabilise the head.Well. Enter the formatter. An additional head is loaded on the outer rim of the drive. Speed control of the motor doesnt even work as we need to measure interval of servo wedges... Which have not been written yet.So the formatter writes a single 'ping'. The motor controllr is told :disengage your servo , go in open loop and simply drive based on your crystal frequency.The formatter now reads the ping and measures the time between two pings. This gives it accurate rotational speed (crystals drift enough so motor speed is off enough to have an impact on data retrieval). The formatter now tunes speed up or down by writing control register in the motor controller. Once the ping is seen at the required rate a square wave is now laid down. Every 'click' (o e to zero or zero to one) marks a potential sector change( not all tracks have the same amount of sectors. Track on the outside are longer than tracks on the inside )We now have a precise physical marker coming from the disk that can tell the electronics : write now.The formatter now grabs the headstack mechanically and moves it using its serve mechanism with anoptical encoder. So this is like a stepper motor with very fine precision. We walk the tracks and everytime there is a tick detected we write a servoburst and the track/sector datablock.Ance this is done we can retract the head that was reading the ticks, close that opening, decouple the headstack and close that opening as well. The drive is now sealed and ready for surface test.The servo mechanisms now all work : the servoburst controls both speed of the motor and timing of write and read operatins and the datablocks tells us where we are. Amplitude detectors keep us centered on the block. Now we write the remaining empty data with a pattern (not just all zeroes... We need changes there so we can do amplitude detection to find the center of the track)Anyway. Like i said, i can go on and on about this. There are now event techniques where a formatter is no longer needed. The electronics that recovers data is so smart and adaptive that speed fluctuations are not a problem. It can adapt at will. Think of it as a serial port where the baidrate should be 9600 but can drift from 9000 to 10000 at random without data corruption. That is what the signal coming from the head looks like. The data samplers can adapt on the fly. You only need a track position but that can also be done because now we have velocity control on the voicecoil.

author: KK
author_rank: Regular Contributor
timestamp: July 05, 2014, 08:05:22 pm
content: ^^^ thanks for that insight. Very interesting.

author: mariush
author_rank: Super Contributor
timestamp: July 05, 2014, 08:19:31 pm
content: (this is going a bit off topic but I'm genuinely curious)free_electron, I've often wondered why hard drive manufacturers wouldn't design a hard drive that would have a head stack on each corner of the unit, so four in total. Just read either four tracks at the same time or a single track using all four heads and assemble the data into the cache of the drive in a continuous segment...I'm thinking this may allow slower rotation speeds which may improve reliability and reduce the heat generated.. or am I wrong about these?Also, is there something physically or otherwise problematic with moving back to 5.25" format with todays technology, let's say in order to make a 10 TB hard drive using 4400-5400 rpm and several platters for companies that want drives for storage? Is it just too hard to make such large platters or it's just considered old technology, they think it wouldn't sell? Or it would be too expensive to change factories to make it when ssd drives are gaining ground?---mojo_chan : hard drives fail in 3-5 years even without keeping them on 24/7. I run my system 24/7 on and have four drives in the system, oldest is about 4 years old and works fine. I replaced one recently due to starting to develop bad sectors though.Here's another article with statistics regarding hard drives and other hardware, from a large french retailer :http://linustechtips.com/main/topic/108284-huge-list-of-failure-rates-on-pc-components-french-but-i-translated-nearly-everything/Point is these people don't all keep their systems on 24/7 and still have hdd failures (preemptive comment: yes, we may not know how those people that return hardware treated those drives, they could have punched the pc or dropped the drives on floor for all we know).I agree with your on just the comment about lightning strikes or fires.. that's still a risk.

author: madires
author_rank: Super Contributor
timestamp: July 05, 2014, 08:29:55 pm
content: Quote from: mojo-chan on July 05, 2014, 08:16:07 pmHave you ever tried to restore 1TB of data from a cloud based backup? I would prefer to restore from a local disk (in an USB box). But the larger problem is to transfer those 1TB to three different cloud storage services. Or do you got a STM64 or better for internet access?

author: David Hess
author_rank: Super Contributor
timestamp: July 05, 2014, 08:47:01 pm
content: Quote from: free_electron on July 05, 2014, 06:20:18 pmSAS drives may have a specific command for it but SATA drives which will reallocate on read may be scrubbed just reading the drive. Even my oldest SATA RAID controllers and drives support this.QuoteDedicated servo designs have been gone for a long time. I just meant to point out that back then, spare sectors were distributed across the disk surfaces to support reallocation and I am sure the same is still the case.QuoteThe part I always found interesting is the economics. Formatting and testing takes time so the only difference between two different drives with different capacities may be the time spent on the machine which writes and verifies the formatting.Semiconductors are often the same way. The difference between two different operational amplifiers may only be the time to test them.

author: rob77
author_rank: Super Contributor
timestamp: July 05, 2014, 08:52:14 pm
content: btw... don't believe to much in cloudsthey're promissing a shitloads of 9s in availability, but that's just marketing... their technology is something like 99% or slightly better (like 99.5% upto 99.9%) and they're simply promising 99.99% or even 99.999%. that's marketing not real availability or real protection of your data, they're simply taking the risk because usually you're achieving 100% during a month in case of no issues. (it's ALL of the cloud services like that).i'm not saying cloud is a lie, i'm saying don't overly stick to the marketing infoif something wrong happens and you'll lose your data, they'll refund you the monthly fee (or part of it) but your data will be gone. so it's better to combine the cloud storage with a local mirrors if you really want to not lose your data.

author: rob77
author_rank: Super Contributor
timestamp: July 05, 2014, 09:11:03 pm
content: Quote from: mojo-chan on July 05, 2014, 08:56:21 pmif my data is lost then i do careand it happens that storage arrays fail and some data is lost (regardless of the raid levels used). actually i witnessed a case when a storage replication between 2 big storage arrays (supposed to increase the data protection level) caused the actual data loss (bug in the firmware). so your data is not 100% safe in the clouds even if they do daily tape backups (you still might lose data for the last 24hours).

author: David Hess
author_rank: Super Contributor
timestamp: July 05, 2014, 09:13:02 pm
content: Quote from: mariush on July 05, 2014, 08:19:31 pmTrack pitch is too tight for them to remain in alignment on different surfaces. This is also why tracking uses an embedded servo signal instead of a dedicated surface. If the actuators were independent this would be possible but the gain is small because of access time and latency. It is more economical to just use an additional drive.QuoteAccess time and latency would suffer. I suspect mechanical runout would limit density as well. Quantum tried this for a while with their Bigfoot series of drives but found it uneconomical. I remember a laptop drive manufacturer trying this with 3 inch drives which could home more than the common 2.5 inch drives yet still be used in a laptop form factor.

author: XOIIO
author_rank: Super Contributor
timestamp: July 05, 2014, 09:13:03 pm
content: Cool, 10TB free, probably not going to fill that up a long time, hopefully it counts for a few years XD

author: David Hess
author_rank: Super Contributor
timestamp: July 05, 2014, 09:19:02 pm
content: Quote from: mojo-chan on July 05, 2014, 08:55:06 pmConner made a dual actuator drive called Chinook but it was not economical.Parallel reading of the heads is not possible because of track misalignment between surfaces. There is even a head setting time when switching heads between surfaces of the same platter.

author: madires
author_rank: Super Contributor
timestamp: July 05, 2014, 09:45:53 pm
content: Quote from: mojo-chan on July 05, 2014, 08:49:17 pmYour cloud storage provider may go bankrupt any time, shut down services, be hacked or face a major disaster. Do you remember the major outage at Microsoft with a broken Hitachi SAN? Several 100,000 of mobile phone users lost their mobility services for several days. Some lost their entire data. Or take this forum for example. How often did it face an outage caused by power or network issues? Large datacenters had total outages, even the backup systems failed. And it will happen again, anytime, anywhere. It takes a lot to design and build a network service with an availability of 99.9% or more. But that doesn't imply that the service will offer that reliability over time. Maintenance procedures may change, failover testing may be never be done or not be part of a regular maintenance schedule. Another datacenter had a power outage and the backup power supply failed too,because it wasn't upgraded to support more servers. The initial deployment was designed for much less servers. "We have to cut down the budget, so let's build that for 30% load and extend it later on", the management said. Who needs capacity planning? When the complete power outage did happen, they knew what they had forgotten.

author: madires
author_rank: Super Contributor
timestamp: July 05, 2014, 09:50:05 pm
content: Quote from: David Hess on July 05, 2014, 09:19:02 pmPlease see WD Black, e.g. WD2002FAEX.

author: David Hess
author_rank: Super Contributor
timestamp: July 05, 2014, 10:23:59 pm
content: Quote from: madires on July 05, 2014, 09:50:05 pmDual actuators as Conner used them are completely different. The Chinook included two completely separate sets of voice coil actuated head assemblies on opposite sides of the platters so a pair of reads or writes could be conducted simultaneously on two areas of the disk. This also had the benefit of halving rotational latency.http://en.wikipedia.org/wiki/Conner_Peripherals#Performance_issues_and_the_.22Chinook.22_dual-actuator_driveThe dual actuators that Western Digital uses are located on the same head assembly and raise the resonate frequency at the end of the head assembly allowing a higher bandwidth for better track following. As far as I know, it does not allow reading or writing to multiple tracks at once and the maximum transfer rate of the drive is not increased beyond that of a single head.

author: free_electron
author_rank: Super Contributor
timestamp: July 06, 2014, 04:36:09 am
content: Quote from: mariush on July 05, 2014, 08:19:31 pmThat is an old idea and has actually been tried , not with four but with two headstacks at opposite corners. You cant do four courners as the heads would bang i to each other the arc on which they move is made very large as to keep the head perpendicular to the track at all times. Now that we have dual actuators this could actually be attempted.But there are other problems. You still have to wait for data to fly by underneath the head. So you could not use both heads simultaneously. So throughputwise you gain nothing. You could maybe seek a bit faster. While stack 1 is reading stack could be seeking the next block. But the amount of mechanics and the associated cost and failure modes make this unusable. Electronics cost nothing. 90% of the cost of a drive is mechanics. Of that cost 30% is the media (platters) 60% the headstack and its actuator and the rest the motor and case.In short : you gain nothing and only drive up the cost. Other factor is the required physical room for a second headstack. This means smaller platters thus less capacity. Its a loose loose situation.Dont confuse dual headstack drives with dual actuator drives. The dual actuator is a second 'motor' o. The headstack that can flex the head.Here is the problem : the heads are at the end of an arm. This arm moves around a fixed point. So the heads essentially move in an arc. If you position the heads in the center of the usable space between hub and rim of the platter , the heads are perfectly positioned on a track. Move the head outwards or inwards from this center and the head now sits under an angle on the track. This the track will be wider !So what we do is mount a piezo element at the tip of the arm where the head is. By flexing the piezo we can twist the head position so it becomes perpendicular again.Imagine it like this.: stretch out your arm , hand flat , fingers pointed out in fro t of you. Draw parallel lines with your arm. Your fingers are perfectly aligned with the line in front of you. Move your arm left or right without bending wrist or elbow. Your fingers now no longer sit parallel with the lines, they sit under an angle. Flex your wrist to adjust your hand. That is what the second actuator does. It gives the head a 'wrist'.This is done to make the tracks uniform in width thus upping the total amount fo data we can cram on a platter.

author: vk6zgo
author_rank: Super Contributor
timestamp: July 06, 2014, 04:59:21 am
content: Thinking about tape storage:-Analog video tape suffers badly from "print-through" from layer to layer after around 20 years.I'm not sure how this would affect error rate of digital signals,though---the first generation of digital VTRs will be coming up on the magic 20 years about now,so that may give you some idea,although the format is probably different.If you really want it to last,convert it to 35mm film stock,or even 70mm.Film from 50 + years ago is still viewable,so there you are!All providing, of course,that there is still anyone with the skills to make a movie projector in 50 years time!

author: rob77
author_rank: Super Contributor
timestamp: July 06, 2014, 07:29:33 am
content: Quote from: vk6zgo on July 06, 2014, 04:59:21 amregarding the magnetic tapes - yes the format is different for LTO tapes... and manufacturers are promising only approx 10 years for a reason (some of them more - that's why i mentioned 10+ in my prev. post). and need to mention again - the 10years is valid only when stored in a controlled environment (temperature, humidity, dust-free). if you need to retain your data for more than a decade then a regular refresh to new tapes is the solution

author: Rigby
author_rank: Super Contributor
timestamp: July 06, 2014, 12:17:45 pm
content: Hard drives are fascinating.I've always wondered why we're married to quarter height drives in desktops these days. Most PC cases I've used could easily mount a half or full-height drive easily, and it seems it would be an easy way to immediately double or quadruple capacity.I saw a 10Mb drive once that was huge and had I think 8 platters and fixed heads everywhere. Nothing moved except the platters. I know that wouldn't be possible today, it was just neat.

author: Shock
author_rank: Super Contributor
timestamp: July 07, 2014, 06:45:12 am
content: This...Buy some eSata connectors if you don't have them. Backup to disk by booting the PC with external disks connected. Use two different reliable brands. After backing up complete a SHA256 read verify against the original data and leave a copy of the checksum on the disk and the PC before shutting down and removing your external disks.Keep one on premises and one off premises and every few months set a reminder to run a verify against the original SHA256 hashes on separate PC'sIf you have problems with a disk do not attempt to verify another disk on that same system. After the original backup keep both sets of hardware isolated to avoid accidental corruption by hardware and software etc.Other things you can do to add reliability:Buy them straight out of the box at the store (avoid mishandling)Updating the firmware of the disksDon't use an external enclosuresStore them an antistatic case or bagStore them where they won't be handled or droppedWhen the disks are new use a utility to read the SMART data from the disk then fill and zero them a few times then compare the SMART data again and evaluation their condition. You can also use utilities to perform more aggressive tests.

author: EEVblog
author_rank: Unknown
timestamp: July 07, 2014, 06:51:47 am
content: I've now got two new 2TB drives, one WD, one Seagate and currently archiving all my raw video onto those. Doing about 65-70MB/sec still takes 7-8 hours per drive.Will then store in separate locations at lab and home.I've also got a new Bluray burner coming and will also burn one copy onto those just to be sure. They will be handy for regular access if I need it. Will try and get some quality TDK or some other brand discs, but they don't seem to be cheap.

author: EEVblog
author_rank: Unknown
timestamp: July 07, 2014, 06:53:05 am
content: Quote from: cleaningOut on July 07, 2014, 06:49:28 amThe raw files are not on youtube. Only a fool would spend 5 years collecting it all and then just throw it all away. You never know when you might need it for something you haven't even thought of yet...Yes, if I lost it the world wouldn't end, but it's certainly worth some effort to preserve.

author: XOIIO
author_rank: Super Contributor
timestamp: July 07, 2014, 07:13:02 am
content: Quote from: cleaningOut on July 07, 2014, 07:09:10 amOh boy, here we go, parenting genius here now.FYI, you don't need to sit at the computer while you are copying the files.

author: EEVblog
author_rank: Unknown
timestamp: July 07, 2014, 07:18:24 am
content: Quote from: cleaningOut on July 07, 2014, 07:09:10 amHe's at Pre-school as I do this, so no. And it's not a huge amount of time, I don't have to sit there watching the files copy.You can say that about every thing I do in my life, including this forum.You forget that this video is my career, and content I have worked hard for 5 years to produce. Are you saying it's not worth me spending a few hundred bucks and an hour or two of my time to archive it all?, that I should just throw it all away?

author: EEVblog
author_rank: Unknown
timestamp: July 07, 2014, 12:08:17 pm
content: Quote from: mojo-chan on July 07, 2014, 08:32:23 amIn my case I only have 500GB/month allowance at the lab, and 250GB/month at home, and both of them include uploads.I just shut down my old ADSL at the lab that had unlimited (but slow) upload.

author: dfmischler
author_rank: Frequent Contributor
timestamp: July 07, 2014, 12:48:45 pm
content: Do you think it's best to leave an archival hard disk on the shelf untouched and untested until needed? Or does it need to be exercised periodically to maintain bearings/grease/seals. Or just periodically copy it to a new disk?An interesting discussion about hard disks and archival media from another forum from a few years ago.Does anybody know of any useful research to try to nail this down (I realize it's a moving target, but even old data would be better than none)?

author: madires
author_rank: Super Contributor
timestamp: July 07, 2014, 12:55:56 pm
content: Quote from: mojo-chan on July 07, 2014, 08:32:23 amExactly! We should say that load and clear, and more often! A lot of people store their data just on a single drive or believe the marketing departmant of the cloud storage provider.QuoteDo you think a company would give you 10TB for free? Wait until you know what you'll have to pay for that free storage.

author: EEVblog
author_rank: Unknown
timestamp: July 07, 2014, 12:57:14 pm
content: Quote from: dfmischler on July 07, 2014, 12:48:45 pmAs a matter of course I'd probably power them up once a year to check. And probably replace after 5 years or so?The cost to replace them isn't much in the scheme of things.

author: rob77
author_rank: Super Contributor
timestamp: July 07, 2014, 01:29:24 pm
content: Quote from: EEVblog on July 07, 2014, 12:57:14 pmsounds goodbut i would keep 2 sets of those drives (2 copies stored separately) . and actually i would probably go for less than 5 years - something like 3 years (but i'm a bit paranid)

author: David Hess
author_rank: Super Contributor
timestamp: July 07, 2014, 07:45:23 pm
content: Quote from: mojo-chan on July 07, 2014, 02:14:30 pmThe US is rapidly catching up on two fronts; allowances are more often being enforced or going down while data demands are rising. One good aspect though is that congestion at the local distribution level is also rising so bandwidth is limited during peak hours anyway.

author: Rigby
author_rank: Super Contributor
timestamp: July 07, 2014, 07:53:05 pm
content: Quote from: David Hess on July 07, 2014, 07:45:23 pmI have a 2000GB monthly quota (up & down) and I never even approach that limit, even when I go crazy on torrent. In my area this is a common quota, and I'm not sure about the rest of the US. My last connection had 150GB/month and I exceeded that every single month, even though the connection was about 15th the speed I have now.

author: pickle9000
author_rank: Super Contributor
timestamp: July 07, 2014, 08:15:38 pm
content: @DaveOut of curiosity roughly how many gigs of files for say 30 minutes of YouTube video (raw and otherwise to be backed up)?

author: mariush
author_rank: Super Contributor
timestamp: July 07, 2014, 08:35:00 pm
content: Cameras record in full HD at about 17-28 mbps (varies from camera to camera, mine's best preset is 28mbps for 1080p 60fps). At about 24mbps average and including sound, you're looking at around 3 MB/s or about 11 GB per hour of recorded video.A 1 hour Youtube video can easily have up to 3-5 hours of content, takes, separate audio recording etc so it adds up.I recently did a 3 part series about temperature measuring with an AD595 and a microcontroller, about 2 hours of actual camera shot video got on youtube from about 3.5 hours of actual footage ... the folder with raw recordings 46 GB.

author: Richard Crowley
author_rank: Super Contributor
timestamp: July 07, 2014, 09:23:28 pm
content: Shooting ratios of 10:1 were not uncommon even back in the days of film. And with video, producers have become even more profligate. Shooting ratios for some types of productions (like documentaries) may approach 100:1. Several multi-camera productions that I work on use ("iso recording") where we are recording each camera, in addition to recording the "live-switched" program.

author: madires
author_rank: Super Contributor
timestamp: July 07, 2014, 09:26:21 pm
content: Quote from: mojo-chan on July 07, 2014, 02:17:25 pmI just looked up the end user price for a WD40EFRX (4TB, 24x7, SOHO NAS/AV usage), it's about EUR 150. That would be one of the cheapest 24x7 disks on the market (but you wouldn't find such disks in a SAN). We assume that we need 4 disks for 10TB and some redundancy. So the disks are EUR 600. They will run about 3 years, i.e. EUR 200 per year. We also have to add the SAN, datacenter, maintenance, spares, internet traffic, new Ferrari for the boss and what have you. Let's say that would be EUR 2300 a year. That means the cloud storage provider pays EUR 2500 a year to offer you 10TB for free. That would be a nice DSO, and you would get another one next year. All for free, just by watching some ads?

author: mariush
author_rank: Super Contributor
timestamp: July 07, 2014, 09:36:18 pm
content: They probably resort to the same mentality as shared hosting... they know few will use more than a few GB of backup and if someone uses more than a certain limit, they can start looking into the account and just say they close the account because of storing copyrighted content or something like that.Other companies like backblaze use their own custom cases and use software that works similar to rsync, storing only differences between files.. see blog post about their storage pods:http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

author: pickle9000
author_rank: Super Contributor
timestamp: July 07, 2014, 10:04:08 pm
content: Quote from: mariush on July 07, 2014, 08:35:00 pmIt's not surprising (the size), the amount of effort that goes into a 2 hour production is.I'm an always amazed at people complaining about backups and the cost of hardware. My reply was always you invest years gathering that information, it is your business. How much would it cost to replace it if it was lost? Would that be the end of your business? I wasn't a computer guy, production testing and setup, but still it's a part of being in business.

author: pickle9000
author_rank: Super Contributor
timestamp: July 07, 2014, 10:13:39 pm
content: Quote from: Richard Crowley on July 07, 2014, 09:23:28 pmI had a small amount of contact with local production cameraman 10 years ago. The shooting ratio is crazy. It is fun to watch productions being shot (not uncommon). Stargate often came out to a nearby lake and provincial park to film/shoot. It's very weird to the Stargate set up in a farmers field and a golf course just 100 feet away.

author: SeanB
author_rank: Super Contributor
timestamp: July 07, 2014, 10:47:05 pm
content: If you use pro photographers you might use around 5 photos from a shoot, but the photographer would have taken possibly 1000 photos and then developed the negatives only. That got rather expensive on 70mm film.

author: Richard Crowley
author_rank: Super Contributor
timestamp: July 07, 2014, 10:55:16 pm
content: Quote from: SeanB on July 07, 2014, 10:47:05 pmThe difference between a professional photographer and an amateur is that the amateur shows you ALL their photos.

author: SeanB
author_rank: Super Contributor
timestamp: July 07, 2014, 11:04:43 pm
content: The Pro only shows you negatives, then you select those that have promise and make a positive transparency of them. Then the final ones are blown up and edited to remove the inevitable marks.

author: Shock
author_rank: Super Contributor
timestamp: July 07, 2014, 11:18:52 pm
content: Quote from: EEVblog on July 07, 2014, 06:51:47 amDave don't bother with Blu-ray. It's as useful as tape drives which are totally a waste of time and expense. Instead buy a 3rd disk in a usb or eSata caddy if you have to.If you refer to my last postyou also need to perform a proper verifies. Because backing up data is fine but you need to confirm it can be read byte exact to the original. This needs to be repeatable over the life of the drive.By using eSata you get the fastest backup and verify (read) speed you can.Once you have sha256 hashes of the original data you no longer need to read it again. You just read the backups.You can use a utility like thePowerShell File Checksum Integrity Verifier (PsFCIV).So no need for a backup program, all they do is add unnecessary complication and things to go wrong (unless you're backing up online databases).

author: EEVblog
author_rank: Unknown
timestamp: July 07, 2014, 11:22:59 pm
content: Quote from: mariush on July 07, 2014, 08:35:00 pmI record at 17Mbps as a compromise. Used to be 12Mbps.

author: EEVblog
author_rank: Unknown
timestamp: July 07, 2014, 11:25:47 pm
content: Quote from: mariush on July 07, 2014, 08:35:00 pmI usually use about 80-90% of my raw material. A good lot of that is simply dead time at the start of the clip.You record audio separately?

author: David Hess
author_rank: Super Contributor
timestamp: July 07, 2014, 11:45:41 pm
content: Quote from: mariush on July 07, 2014, 09:36:18 pmI remember studying what Backblaze published carefully when I looked into doing this for myself on a much smaller scale. This article discussing various storage controllers is informative as well:http://blog.zorinaq.com/?e=10

author: pickle9000
author_rank: Super Contributor
timestamp: July 07, 2014, 11:51:23 pm
content: Quote from: Richard Crowley on July 07, 2014, 10:55:16 pmI thought that was called "being a grandmother".

author: apelly
author_rank: Unknown
timestamp: July 08, 2014, 12:19:50 am
content: Quote from: Richard Crowley on July 04, 2014, 01:17:30 amMe too. Three anyway. One just yesterday.

author: pickle9000
author_rank: Super Contributor
timestamp: July 08, 2014, 12:42:17 am
content: Quote from: apelly on July 08, 2014, 12:19:50 amAnd thus the need for multiple backups. Hardware, media, fire, storage location, the list goes on.

author: mariush
author_rank: Super Contributor
timestamp: July 08, 2014, 01:30:13 am
content: Quote from: EEVblog on July 07, 2014, 11:25:47 pmNo. I either let the camera record with its own internal microphones (5.1 dolby something) or I plug a lavalier microphone in it and then it switches to stereo recording automatically.For one video, i recorded some segments without lavalier mic so the audio was 5.1. I wanted to have that same 5.1 sound and that same audio processing the camera does when showing the screen capture, so I put the camera on tripod by the monitor and recorded with the camera in sync with the screen capture software. Then I copied video from camera to pc and extracted the audio and threw away the video, so now my screen capture segments had the original stereo sound produced by the desktop capture software (using lavalier mic in the pc soundcard jack) and the 5.1 sound from the camera.I've had segments messed up for lots of reasons .. ambulance driving with sirens behind my apartment building, phone ringing.. not a big deal but if I stop I sometimes decide I could say that "paragraph" in a better way and reshoot the last 5 minutes of an explanation or something like that. It adds up.

author: EEVblog
author_rank: Unknown
timestamp: July 08, 2014, 06:41:22 am
content: Quote from: pickle9000 on July 07, 2014, 08:15:38 pmFYI, my latest 29min edited LVDC video is 4.86GB of raw video, with a total length of 39:46 over 53 clips.There were a few clips in that were nothing and I didn't use. I don't bother deleting those in-camera at the time because it's just too much hassle to go into playback mode, select delete mode, switch back, and then reset my exposure setting (annoying bug in the Canon HF G10). Nor do I delete when nothing clips on the PC, it's just not worth the effort.

author: pickle9000
author_rank: Super Contributor
timestamp: July 08, 2014, 07:08:02 am
content: Quote from: EEVblog on July 08, 2014, 06:41:22 amTo be honest that seems ridiculously good, both in raw video and shooting ratio.

author: EEVblog
author_rank: Unknown
timestamp: July 08, 2014, 07:42:20 am
content: Quote from: pickle9000 on July 08, 2014, 07:08:02 amYeah, I think it's pretty good. Most of what comes out of my mouth ends up on camera.If people are really keen to see that in action, I've posted some complete video of me shooting an entire mailbag video.I don't fuss that much over the details, and I also rarely replay clips before I go edit. And also rarely go back to reshoot something after I watch it in the edit.

author: EEVblog
author_rank: Unknown
timestamp: July 08, 2014, 08:10:14 am
content: Quote from: wilfred on July 08, 2014, 08:02:26 amYes, I've mentioned it a few times it's something I want to do.Just the methods of shooting/editing/tools etc would take several videos.Not easy to shoot either because I'd have to setup multiple camera shots etc. e.g. to record how I record. Showing the effects of various settings etc. Not easy stuff to show that well.The monetisation/comments thing would be another video again.

author: SirNick
author_rank: Frequent Contributor
timestamp: July 08, 2014, 08:37:58 pm
content: Blu-ray is fine. Will it last 20 years? Probably not. Some of the CD-Rs I burned on my first 4x writer are still readable, but things change. Regardless, if you're not putting all your eggs in that basket, it's still a convenient way to store and catalog data for easy retrieval. I envision something where each disc is labeled "raw footage from episode #327" or whatever. If the disc fails, fine, go to the HDDs. It's inexpensive to add an internal (or USB) reader to any computer, assuming it doesn't already have one, so accessibility is good.As to HDD reliability, I have a whole stack of 40 and 80GB drives. One developed bad sectors last time I tried to read it (about a year or two ago.) Others still work and hold, for e.g., Hackintosh HDD images (because it's a laborious and painfully slow process to reinstall). If I lose them, ... eh.I have 4 drives (RAID-5) in a NAS. When I replace those for bigger capacity drives, the old 4 become backup drives for the new ones. NO RAID on the backups -- you don't want the hassle of trying to arrange for an appropriate read environment later on. A USB-to-SATA cable or dock will be available for at least as long as the drive is viable. This process of upgrade and trickle down keeps each generation of drives "fresh" within 2-5 years. I plug them in every six months or so to backup new data. (It really should be more often, but that's a risk I take.) IMO, this is a decent compromise between leaving them at rest, and periodically checking for faults.Since video archival won't be updated, doing a checksum validation of backed-up data once a year would suffice to catch drive faults or media read errors before both the BD-R and HDD have had a chance to fail. Could a simultaneous failure still occur? Yes, of course. That's life. Dave might be out one episode of raw video then. I suspect he'll survive.

author: ve7xen
author_rank: Super Contributor
timestamp: July 08, 2014, 11:38:20 pm
content: If I were doing this:- Replace commodity NAS with FreeNAS or NAS4Free running ZFS in a redundant mode as primary archive. Schedule regular scrubs. Comes with free replication to a remote server, if you decide that's feasible, plus block-level checksums and other useful features. Since this is fairly cheap, I would probably build a duplicate box at home and use USB media to carry the incremental snapshot home with you after you finish an episode. Don't remove your snapshots and you're safe from pretty much everything but simultaneous and catastrophic hardware or filesystem failure, and you have live access to all your data at any time.- Secondary archive to BD with par2 or dvddisaster, keep in fire safe / safe deposit box / with data archival company- Upgrade your network to GigE FFS!

author: SirNick
author_rank: Frequent Contributor
timestamp: July 09, 2014, 12:15:39 am
content: Ethernet speed isn't necessarily the bottleneck. I typically get ~30MB/s over CIFS from a Win 7 desktop to my NAS. All Gig-E. But, the NAS is RAID 5 on JFS, all in software on a Mobile Core 2 Duo mini ITX motherboard.Some of those little toaster NASeses are running at a few hundred MHz, and may or may not have any hardware acceleration. Gig-E is merely a bullet point for marketing for all the good it's going to do you.

author: ve7xen
author_rank: Super Contributor
timestamp: July 09, 2014, 12:45:23 am
content: Quote from: SirNick on July 09, 2014, 12:15:39 amYou should be able to do much better. Any remotely modern hardware (which this is) should be able to saturate GigE if set up properly. I do software RAID6 and can get ~300MB/s read and 150MB/s write out of the array (which is poorly tuned for block size and stride). On NFS it saturates the GigE in either direction, but CIFS required a lot more messing around to work well and still doesn't match it for performance. Many times people complain about less than ideal performance it's due to using crappy Realtek NICs on one or both ends.QuoteOh for sure, 10MB was just a "suspicious" number.

author: SirNick
author_rank: Frequent Contributor
timestamp: July 09, 2014, 01:15:45 am
content: Definitely. I have no doubt it can be improved dramatically. I tried "dd if=/dev/zero of=/mnt/md" after I built it, and got significantly higher throughput than what I see from across the LAN. I don't remember what it came out to... pretty sure it didn't even come close to a couple hundred MB/s, but it was obvious that performance over the network -- including and especially CIFS -- is a constraint.That said, throughput is so rarely a concern that I haven't bothered to pursue any of the bottlenecks. Can stream 1080p? Check. Decent chance of saturating a single-spindle source or destination? Check. OK, good enough then.And of course there's an excellent chance that both ends use Realtek chipsets, as both the NAS mobo and the PC are using onboard LAN. That's hardly the worst sin though. It's also stacked up on top of a cabinet in the laundry-slash-boiler room, keeping those 1U PSU CapXon electrolytics cooled by 25-30C ambient temps.Wouldn't be caught dead implementing a system like that for a business, but at my own home? Eh. "Good enough for Australia."

author: mariush
author_rank: Super Contributor
timestamp: July 09, 2014, 01:23:25 am
content: QuoteIt depends. For example, the $1000 NAS I linked to a couple of pages or so before ( Synology DS1813+ ) has 4 gigabit ports that can be linked together for a 4 gbps throughput. If you have a smart switch capable to do port trunking, you're in business.That NAS uses 2.13 Ghz dual core Atom...ps. You can spend 400$ on a hardware raid card with 8 sata ports like this one :http://www.newegg.com/Product/Product.aspx?Item=N82E16816151137Add ~150-200$ on cpu, mb, case and ram and optionally get a 2-4 port gigabit network card from eBay and do port trunking. The software's going to be a problem though, there is freenas and other ready made solutions but it's still not going to be as polished as that Synology comes from the factory.

author: Rigby
author_rank: Super Contributor
timestamp: July 09, 2014, 01:42:45 am
content: Ethernet throughput matters a lot on what is copied.Lots of small files will not saturate gig-e. A few large files will saturate any link. There is an overhead per file that impacts transfer speed in any per-file upload/download/copy.To maximize throughput, zip up the files with a non-compressing archive format, and transfer them in that form.

author: Legit-Design
author_rank: Frequent Contributor
timestamp: July 09, 2014, 02:12:01 am
content: http://www.smallnetbuilder.com/lanwan/lanwan-features/30201-need-to-know-jumbo-frames-in-small-networksOne word, Jumbo frames

author: mariush
author_rank: Super Contributor
timestamp: July 09, 2014, 02:34:08 am
content: Way to go, posting an article last updated in 2007 and a graph showing cpu usage on a300 MhzSun server.Now a dual core i3 can probably run in circles and saturate 1gbps even without jumbo frames.

author: SirNick
author_rank: Frequent Contributor
timestamp: July 09, 2014, 03:10:20 am
content: Quote from: mariush on July 09, 2014, 01:23:25 amYou can, and (IMHO) that would be a waste of $400.I have a rack chassis with 4 bays. I have a motherboard with four SATA ports (and a CF card slot). Connectivity is covered."But what about hardware-optimized RAID then?" Good point, Bob. But where you see a purpose-built ASIC that increases IOPS, I see a data striping algorithm that requires the original hardware to recover data from my disks. I once had a server with a proprietary controller card that made RAID really fast. Then the card died, and it was NLA, and I no longer had access to the data on those disks."But what about backups?" Yes, Bob, I had backups. After a few hours of rebuilding the server to the point where the backup software client could be installed, it only took another hour or so to get the data back. Or, you can pull a commodity PC off the shelf, plug in the NIC and power cables, stuff the hard drives in there, and have any Linux Live CD recognize the MD RAID volume and you're back online in 15 minutes.On another note: I once did something colosally stupid and offlined two of the four disks in my RAID-5 set. An ordinary controller would have dumped the volume at that point. But, MD RAID just complained furiously until I gave it the "dash F" option, which stands for "Shut the dash-F up and re-mount the volume regardless". After replaying the journal (and running a background integrity check), it was 100% accessible. No lost data. I am a big, big fan of Linux MD RAID.

author: ve7xen
author_rank: Super Contributor
timestamp: July 09, 2014, 04:07:34 am
content: Quote from: mariush on July 09, 2014, 01:23:25 amFor the task at hand, I would say hardware RAID is a big no-no. It's expensive, you're not targeting high IOPS, and now your data depends on having a working RAID card of correct model and firmware. You can get an 8-port SAS HBA for under $100, if you even need one, since the 6 or 8 onboard SATA ports common these days are likely to be sufficient, at least to start.This guydid it recently with a pretty nice build for around $400 sans-drives. I would personally throw in an Intel PCIe NIC just because onboard NICs suck, but hardly required.Also keep in mind that port-trunking/link aggregation chooses a link based on a hash of some portion of the header (often just the source/destination MAC addresses, though sometimes layer 3 or even layer 4 on higher-end gear). In other words, you're primarily concerned with throughput to a single client, this won't help at all, since all the traffic will end up on the same link.Nas4Free is pretty polished these days, and gets you ZFS, which is hands down worth any slight increase in effort required over a commercial product. Automatic scrubbing and block checksums are exactly what you need for long-term archival. Snapshots, and ones which can be incrementally copied to a remote system by sneaker- or Inter-nets would also be pretty handy. Most of the commercial boxes are based on standard Linux technology, so you can plop the disks into any machine to recover your data if the NAS box goes tits up, but a nice benefit of building on Nas4Free/FreeNAS/Linux is that youknowyou can do this in a pinch and that the vendor isn't doing something weird (like Drobo and presumably others).Edit: Ninja'd but I'll leave the post anyway

author: Legit-Design
author_rank: Frequent Contributor
timestamp: July 09, 2014, 05:17:23 am
content: Quote from: mariush on July 09, 2014, 02:34:08 amnice video with real life test.That NAS box probably doesn't run processor faster than300 Mhzsomething? It all depends on situation.In my opinion it's a fair comparison of almost decade old server vs modern NAS box.Just found this quote:Quote from:https://superuser.com/questions/346717/should-i-enable-jumbo-frame-size-of-9000-bytes-for-my-nics

author: rdl
author_rank: Super Contributor
timestamp: July 10, 2014, 03:01:06 pm
content: Any time the discussion of backing up data arises I am reminded of this video.