Migrating data

We compare several methods for copying/migrating large amounts of data. The aim is to preserve all file permissions and metadata, which requires root access on both source and target filesystems.
We include information about some options that aren't available on the BusyBox implementation of these tools, because you may be copying data to/from systems with other Unix tools installed, which do provide those options.
== Bit-copying methods ==
If your source data occupies an entire partition (that is, nothing else resides on that partition), and your target partition is the same size or larger, and you're willing to have it use the same filesystem as the source, then one option is to bit-copy the whole source partition:
Boot from LiveCD and make sure that neither /dev/sourcepart nor /dev/targetpart are mounted.
Example:
This also copies unoccupied blocks on the sourcepart to the targetpart. A more optimal variant is to use partimage, which copies only the used portions of the sourcepart. One limitation is that currently (v0.6.9, released July 2010, current as of Mar 2012) partimage doesn't support ext4 format.
Once you've finished, the filesystem on targetpart may occupy less than the total space available on targetpart. You may want to use a tool like resize2fs to grow the partition.
== Higher-level methods ==
The other options we'll discuss assume the target filesystem is already formatted and mounted as . 
If you need to copy volatile data from a running system, you may want to copy from a lvm snapshot, or do the copy from single-user mode, or from a LiveCD or other boot medium so your source data is not in use during the copy. If you want to copy from or such, no such precautions are necessary.
If you're copying a complete filesystem, you'll probably only want to copy the and mountpoints, but not their current contents. Remember to copy and and , the last two of which may require special attention. Pay attention to other mountpoints (such as , ) and RAM-based filesystems (such as and ).
=== cp ===
Depending on which system you're working with, and what kind of data you're copying, you may be able to do a satisfactory local copy using just cp. However, there are a number of limits that prevent this from being a general solution.
The Gnu implementation of cp provides the following options (among others):
* -a: same as -RP -p/--preserve=mode,ownership,timestamps,links,context,xattr
* -v: verbose
* -l: make hard links instead of copying
* -s: make symlinks instead of copying (source file names must be absolute)
* --sparse=always
* -u, --update: don't copy files or symlinks over an existing file unless its mtime is older
* -x, --one-file-system: skip subdirectories on different volumes
 
The BusyBox version provides all of the GNU options except --sparse, -u and -x, and may lack the context and xattr options to --preserve. It will preserve hard links: if and are hard links, and will also be hard links with each other (though not with the originals). BusyBox's cp seems to silently ignore the -v option.
The FreeBSD implementation provides all of these except -s, --sparse and -u, and the --preserve options from links on: in particular, it always copies hard links as separate files. The Mac version shares the limitations of the FreeBSD version, and additionally lacks -x. Also, on the Mac version the -a shortcut isn't available, you must explicitly say -RPp.
=== tar ===
Options recognized by BusyBox and other implementations:
* -c: create a new archive, records existing perms and mtimes, symbolic owner/group (can specify --numeric-owner), hard links among archive elements
* -x: extract files from an archive, uses current owner/group unless root, subtracts current umask unless root or -p, when extracting hard links must include first
* -t: list the contents of an archive
* -f archive: use -f - for stdin/stdout
: 
* -O: extract files to standard output
* -C dir: change to directory dir, when creating this is order-sensitive
* -v: verbose
* -z: compress archive with gzip
* -j: compress archive with bzip2
* -Z: compress archive with compress
* -h: follow symlinks, archive and extract the files they point to (like -L in other programs)
* -m: don't extract file's original mtime, leave touched with the extraction time
* -T file: get names to extract or create from file, which can also include "-C/etc" lines
* --exclude=pattern: exclude specified glob patterns
* -X file: exclude glob patterns listed in file
: 
The following common options are allegedly also honored by BusyBox (according to
comments in its source) but aren't declared in its --help:
* -o, --no-same-owner: (BSD doesn't recognize a long option) extract as yourself (default for ordinary users)
* -p, --same-permissions: (BSD doesn't recognize this long option) extract permissions verbatim, instead of subtracting current umask (usually default for root)
* --no-same-permissions: subtract umask from extracted permissions (default for ordinary users)
* --numeric-owner: create using numbers for user/groups
* -k: don't replace existing files when extracting
--overwrite: (not on BSD) more aggressively overwrite existing files and directory metadata when extracting
 tar's default behavior is to first remove existing files (otherwise, all their hardlinks would be modified) and symlinks.
 If an existing dir is nonempty, its existing contents won't be removed; tar will instead just update the dir's metadata.
 --overwrite instead follows existing symlinks, and removes anything that impedes extraction except non-empty dirs.
 (Gnu's --recursive-unlink removes even those, may replace them with files or symlinks.)
GNU options (Mac and FreeBSD as well, unless noted) include:
* --one-file-system: when creating archive, stay on the volume(s) where the specified roots are located, and don't cross mountpoints
* --null: -T reads null-terminated names, ignores "-C/etc" entries
* -l, --check-links: (BSD for -c only) when creating or extracting, print a message if not all hard links are processed
* -S, --sparse: (not on BSD) when creating archive, handle sparse files efficiently
Some GNU tar formats:
* in the future, it will default to -H posix/pax, which is the POSIX.1-2001 format
* v.1.13 - current (v1.26 released Mar 2011, still current in Mar 2012): defaults to -H gnu, which is not very different from the -H oldgnu format.
* -H ustar is the portable POSIX.1-1988 format, some limits including max 256 char filenames, max 8G filesize; supports special files, but not sparse files. Also seems not to support xattrs and ACLs (other tar formats are meant to handle ACLs, but see comments below)
Example of local copy:
The optional flags are:
* --one-file-system: Stay on the volume where is located. This may or may not be the behavior you desire. Note that this feature isn't available on BusyBox tar in any case, though it is available in some other tar implementations.
* -v: verbose
Example of using tar in cpio-style (for which, see below):
The next examples add the -z flag, to compress output using gzip. If you have a fast connection, you may omit this; alternatively, you may use -j for bzip2. Example of copying to remote machine:
Example of copying from remote machine:
=== cpio ===
cpio operates in one of four modes:
* -o -H newc: create archive on stdout (or -F archive), including filenames supplied from stdin (like tar -cf- -T-)
: -H newc: (on Mac, -H sv4cpio and/or -c) specifies to use SVR4 ASCII format, this is required to create archives in BusyBox
* -i: receive archive from stdin (or -F archive), extract specified patterns underneath curdir (like tar -xf-)
* -t: (on some systems, may need -it) receive archive from stdin (or -F archive), list contents to stdout (like tar -tf-)
* -p /target: receive list of files from stdin (or -F archive), copy them underneath (like tar -cf- -T- | tar -xf- -C /target) 
tar vs cpio:
* if you have includes/excludes, it's easier to use find than the unfamiliar and non-portable options of GNU tar
* cpio handles special files (block and char devices, fifos, etc) traditional tar (-H v7 format) didn't, also didn't store symbolic owner/group
* when extracting hard links, tar must always include the first
* tar and the -H newc cpio format handle 32-bit inodes; though some older cpio formats didn't
* the -H newc cpio format has max 1024 char filenames (though GNU cpio can handle arbitrary lengths) and max 4G filesize. The portable -H ustar tar format has max 256 char filenames and max 8G filesize limits
* in some implementations, cpio wouldn't copy files over 2G properly. Don't have details about which implementations/versions are so limited
* some tar formats are supposed to handle acls, however in practice this doesn't seem very reliable. Not sure what the situation is with cpio. rsync is known to work well for such cases
Example of local copy:
The flags are:
* -xdev: stay on the volume where is located; this may or may not be the behavior you desire
* \! -path ./lost+found: omit empty ; ! only needs to be escaped for some shells, and this sequence may be repeated
* -print0 ... -0: permits processing filenames with embedded \ns, not available on BusyBox cpio or Mac
* -d: create leading directories, like mkdir -p
* -m: preserve file's original mtime
* -v: verbose
* -u: overwrite existing files, even if they're newer than files being copied
* -l: create hard links instead of copying. This may or may not be the behavior you desire. Note that this feature isn't available on BusyBox cpio in any case, though it is available in some other cpio implementations.
Other flags are:
* -oA -F archive: append to existing archive (not on BusyBox or BSD)
* -L: follow symlinks (not on BusyBox)
* --sparse: write files with blocks of zeros as sparse files (Gnu only)
Example of copying to remote machine:
=== rsync ===
rsync must be installed on both source and target machines. Example:
Note that the trailing on is essential, if you want to end up a clone of .
The flags are:
* -v: verbose
* -z: compress data during transfer
* -x or --one-file-system: skip subdirectories on different volumes
* --numeric-ids: rsync's default is to use symbolic user/groupnames
* -H or --hard-links: preserve hard links in copied data, this can be expensive, and won't break existing hard links on unless rsync needed to write updated data to some of the linked files
* -A or --acls: preserve ACLs
* -X or --xattrs: preserve xattrs
* -8 or --8-bit-output: don't escape chars in filename even if they're invalid in current locale
* --partial: keep partially-transferred files
--inplace: overwrite existing files in place
rsync's default behavior is to write new files and move them into place when complete. This breaks any hard links the existing file may have had and makes a copy-on-write filesystem see the file as entirely new. --inplace specifies an alternate behavior, of writing updated data directly into the existing file.
* -h or --human-readable: report statistics in prettier form (use once for powers of 10, twice for powers of 2)
* --progress: show progress of each file transferred, implies -v
* -n or --dry-run
rsync's full manpage
rsync's advantages:
* like cp, can handle ACLs and xattrs. Many tar and cpio implementations don't
* like cpio, can optionally work over a network
* can resume incomplete transfers, and can do incremental updates if the data is being copied multiple times
Including/excluding from transfers:
* --include=pattern[/]
* --exclude=pattern[/]
Each of these may be repeated. The patterns may contain *, **, ?, [a-z].
A trailing dir/*** matches both dir and its contents.
If pattern contains a ** or (non-trailing) /, it matches against the full path from , else it only matches against the final path element.
=== lvm ===
"This is one reason I like LVM. Just add the new disk to the volume group, pvmove the logical volumes from the old to new disk, [wait a bit], remove the old disk from the volume group, and then from the system. If it's your boot disk you're replacing then you also need to update your boot loader." 
See Setting up Logical Volumes with LVM.
=== Other tools ===
Category:Installation
Category:Storage