Parchive

Related articles
List of applications/Security#Hash checkers
Parchive
(Parity archive) is a file verification and repair tool using PAR2 files to detect damage in data files and repair them if necessary. It can be used with any kind of file.
Installation
Install
the
par2cmdline
package. The commands
par2
,
par2create
,
par2verify
and
par2repair
are now available.
How it works
par2create
takes the input file(s) and interprets the input as a certain number of data blocks. Based on the data blocks,
par2create
then creates recovery blocks with the help of the
Reed–Solomon error correction
code. Later, you can trade any recovery block for any corrupted data block in order to repair the source data. You need as much recovery blocks as data blocks have gone corrupted in order to repair the file(s) successfully.
Let us say you want to calculate 30% of recovery information for a precious file:
$ par2create -r30
file
Parchive now has created the
file
.par2
index file which is essentially not needed for recovery. Additionally, it has created the recovery blocks and has spread them into multiple files. If you created, say, 592 recovery blocks, you will find the files
file
.vol000+001.par2
file
.vol001+002.par2
file
.vol003+004.par2
file
.vol007+008.par2
file
.vol015+016.par2
file
.vol031+032.par2
file
.vol063+064.par2
file
.vol127+128.par2
file
.vol255+256.par2
file
.vol511+081.par2
The number left of the plus sign is the index of the first recovery block in the particular file and the number on the right the number of recovery blocks the file provides.
In the early days where no integrity check was done on link level these files proved useful since one could select recovery files based on the number of data blocks gone corrupted. If your download left you with 43 corrupted data blocks, you could convert the number to binary and instantly see what recovery files you had to fetch:
32 16  8  4  2  1
43 = 1  0  1  0  1  1

→ download *+032.par2, *+008.par2, *+002.par2 and *+001.par2.
This is very efficient in terms of bandwidth usage. One would finally call
$ par2repair
file
*.par2
in order to repair the downloaded file(s). You can ignore the index file since Parchive can handle a missing index file. Sometimes it is useful to include the
-N
parameter in order to reach an intact data block which would otherwise go undetected.
Tips and tricks
Batch-protecting your files
It may be the case that you do not want to serve clients recovery files, but want to ensure additional integrity of your files. In times of helium filled hard disk drives, shingled-bit technology, dense capacity per square centimeter, transfer losses etc. the probability of bit rot is high. Remember that you should always have (automated)
backups of your data
, but a little additional protection does not hurt, especially since we have so much storage available today. By creating
par2
files you have a much more convenient way to verify the integrity of your data and restore the data than running application programs over the files and sieving for error outputs. Bit rot now can happen in both the original file(s) AND the recovery file(s), and you still can repair the original file(s).
As a consequence, one recovery file containing all recovery packets is sufficient (parameter
-n1
). This also reduces the amount of recovery data. An important question to answer is the percentage of redundancy you want to have. Especially for smaller files (<1 MiB) the amount of recovery data does not really correlate with the original file size:
Original data
percentage
Recovery data (without index)
184.8 KiB
5
287.8 KiB
184.8 KiB
100
743.1 KiB
3.4 MiB
5
458.8 KiB
3.4 MiB
30
1.5 MiB
3.4 MiB
100
4 MiB
1.7 GiB
5
87.6 MiB
5% is a reasonable amount of recovery data but you can go up to 100% recovery data for really important files. 100% recovery files can restore your file if you accidentally deleted the original one and you are too lazy to search for the file in your backup (you have one, have you? :).
Here is a simple script which runs over the current directory recursively and batch-protects the files:
Tip
You may have to make sure that your
bin/
folder is in the search path:
$ export PATH=$PATH:$HOME/bin/
Put this line in your init file in your home directory to make it permanent.
~/bin/batchprotect.sh
#!/bin/bash
readonly REDUNDENCY_PERCENTAGE=5
readonly PRINT_ON_SUCCESS=1

create_par2() {

  local current_file="${1}"
  if [ ! -e "${current_file}-5p.par2" ] && [ -s "${current_file}" ]; then
    if par2create -n1 -q -q -r"${REDUNDENCY_PERCENTAGE}" "${current_file}"; then

      if [ "${PRINT_ON_SUCCESS}" = 1 ]; then
        printf '\033[1;37m%s \033[1;34m%s \033[1;32m%s \033[0m\n' \
                'Creating par2 on' "${current_file}" 'successful'
      fi

      rm "${current_file}.par2"
      mv "${current_file}".vol*par2 "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2"
      return 0

    else
      printf '\033[1;37m%s \033[1;34m%s \033[1;31m%s \033[0m\n' \
              'Creating par2 on' "${current_file}" 'failed'
      return 1
    fi
  fi
}

verify_par2() {

  local current_file="${1}"
  if [ -f "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2" ]; then
    if par2verify -q -q "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2"; then
      if [ "${PRINT_ON_SUCCESS}" = 1 ]; then
        printf '\033[1;37m%s \033[1;34m%s \033[1;32m%s \033[0m\n' \
                'Verifying par2 on' "${current_file}" 'successful'
      fi
      return 0
    else
      printf '\033[1;37m%s \033[1;34m%s \033[1;31m%s \033[0m\n' \
              'Verifying par2 on' "${current_file}" 'failed'
      return 1
    fi
  fi
}

repair_par2() {

  local current_file="${1}"
  if [ -f "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2" ]; then
    if par2repair -p -q -q "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2"; then
      if [ "${PRINT_ON_SUCCESS}" = 1 ]; then
        printf '\033[1;37m%s \033[1;34m%s \033[1;32m%s \033[0m\n' \
                'Repairing' "${current_file}" 'successful'
      fi
      return 0
    else
      printf '\033[1;37m%s \033[1;34m%s \033[1;31m%s \033[0m\n' \
              'Repairing' "${current_file}" 'failed'
      return 1
    fi
  fi
}

main() {

  while read -r current_file; do
    # Create or verify
    if [ ! -e "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2" ]; then
      create_par2 "$current_file"
    else
      if ! verify_par2 "$current_file"; then
        repair_par2 "$current_file"
      fi
    fi
  # We ignore the file itself, any par2/sig extension and any .git directory and print without the leading ./
  done < <(find . -type f ! -name '*par2loop.sh' ! -name '*par2' ! -name '*.sig' ! -path '*.git*' -prune -a -printf '%P\n' | shuf)
}

main
You would then call
batchprotect.sh
in order to have all files in the current directory recursively protected by Parchive. Additionally, you do not have to maintain list of checksums since gpg can do that for you, too.
Verification
$ par2verify
file
-5.par2
$ cfv
file
-5.par2
$ gpg --verify
file
.sig
And, if you changed the path of the original file:
$ par2verify -B /
file
-5.par2
/new/path/to/fileRenamed
$ gpg --verify
file
.sig
/new/path/to/fileRenamed
You can also change the path of the
par2
/
sig
file.