CVE-2022-4773 (Java, published 2022-12-28T00:15Z)
Description: value: ** UNSUPPORTED WHEN ASSIGNED ** A vulnerability classified as problematic was found in cloudsync. Affected by this vulnerability is the function getItem of the file src/main/java/cloudsync/connector/LocalFilesystemConnector.java. The manipulation leads to path traversal. It is possible to launch the attack on the local host. The name of the patch is 3ad796833398af257c28e0ebeade68518e0e612a. It is recommended to apply a patch to fix this issue. The associated identifier of this vulnerability is VDB-216919. NOTE: This vulnerability only affects products that are no longer supported by the maintainer.
Severity: nan (CVSS v3 base score: 3.3)
Weakness: CWE-22 \u2014 Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal')
The product uses external input to construct a pathname that is intended to identify a file or directory that is located underneath a restricted parent directory, but the product does not properly neutralize special elements within the pathname that can cause the pathname to resolve to a location that is outside of the restricted directory.
Repository: https://github.com/HolgerHees/cloudsync
Files changed:
.gitignore
.project
INSTALL.txt
LICENSE
README.md
cloudsync.bat
cloudsync.sh
config/cloudsync.config.default
pom.xml
src/assembly/dep.xml
src/main/java/cloudsync/Cloudsync.java
src/main/java/cloudsync/connector/LocalFilesystemConnector.java
src/main/java/cloudsync/connector/RemoteConnector.java
src/main/java/cloudsync/connector/RemoteDropboxConnector.java
src/main/java/cloudsync/connector/RemoteDropboxOptions.java
src/main/java/cloudsync/connector/RemoteGoogleDriveConnector.java
src/main/java/cloudsync/connector/RemoteGoogleDriveOptions.java
src/main/java/cloudsync/connector/RemoteLocalFilesystemConnector.java
src/main/java/cloudsync/connector/RemoteLocalFilesystemOptions.java
src/main/java/cloudsync/exceptions/CloudsyncException.java
src/main/java/cloudsync/exceptions/DataException.java
src/main/java/cloudsync/exceptions/FileIOException.java
src/main/java/cloudsync/exceptions/InfoException.java
src/main/java/cloudsync/exceptions/UsageException.java
src/main/java/cloudsync/helper/CmdOptions.java
src/main/java/cloudsync/helper/Crypt.java
src/main/java/cloudsync/helper/Handler.java
src/main/java/cloudsync/helper/Helper.java
src/main/java/cloudsync/logging/LogconsoleHandler.java
src/main/java/cloudsync/logging/LogfileFormatter.java
src/main/java/cloudsync/logging/LogfileHandler.java
src/main/java/cloudsync/model/Item.java
src/main/java/cloudsync/model/ItemType.java
src/main/java/cloudsync/model/LocalStreamData.java
src/main/java/cloudsync/model/RemoteItem.java
src/main/java/cloudsync/model/RemoteStreamData.java
src/main/java/cloudsync/model/TempInputStream.java
src/main/java/cloudsync/model/options/ExistingType.java
src/main/java/cloudsync/model/options/FileErrorType.java
src/main/java/cloudsync/model/options/FollowLinkType.java
src/main/java/cloudsync/model/options/NetworkErrorType.java
src/main/java/cloudsync/model/options/PermissionType.java
src/main/java/cloudsync/model/options/SyncType.java
src/test/java/cloudsync/FilesystemHelper.java
src/test/java/cloudsync/RemoteLocalFilesystemTest.java
Commit message: vuln-fix: Partial Path Traversal Vulnerability

This fixes a partial path traversal vulnerability.

Replaces `dir.getCanonicalPath().startsWith(parent.getCanonicalPath())`, which is vulnerable to partial path traversal attacks, with the more secure `dir.getCanonicalFile().toPath().startsWith(parent.getCanonicalFile().toPath())`.

To demonstrate this vulnerability, consider `"/usr/outnot".startsWith("/usr/out")`.
The check is bypassed although `/outnot` is not under the `/out` directory.
It's important to understand that the terminating slash may be removed when using various `String` representations of the `File` object.
For example, on Linux, `println(new File("/var"))` will print `/var`, but `println(new File("/var", "/")` will print `/var/`;
however, `println(new File("/var", "/").getCanonicalPath())` will print `/var`.

Weakness: CWE-22: Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal')
Severity: Medium
CVSSS: 6.1
Detection: CodeQL & OpenRewrite (https://public.moderne.io/recipes/org.openrewrite.java.security.PartialPathTraversalVulnerability)

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/13

Co-authored-by: Moderne <team@moderne.io>

Vulnerable code:

Fixed code:
.settings*
config/cloudsync.config
lib*
target*
.classpath
.project
/bin/
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
	<name>cloudsync</name>
	<comment></comment>
	<projects>
	</projects>
	<buildSpec>
		<buildCommand>
			<name>org.eclipse.jdt.core.javabuilder</name>
			<arguments>
			</arguments>
		</buildCommand>
		<buildCommand>
			<name>org.eclipse.m2e.core.maven2Builder</name>
			<arguments>
			</arguments>
		</buildCommand>
	</buildSpec>
	<natures>
		<nature>org.eclipse.jdt.core.javanature</nature>
		<nature>org.eclipse.m2e.core.maven2Nature</nature>
	</natures>
</projectDescription>
to use the git clone follow the maven build instruction

https://github.com/HolgerHees/cloudsync/wiki/Maven-HowTo

the binary releases are runnable without any additional steps.

https://github.com/HolgerHees/cloudsync/releases
GNU GENERAL PUBLIC LICENSE
                       Version 2, June 1991

 Copyright (C) 1989, 1991 Free Software Foundation, Inc., <http://fsf.org/>
 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 Everyone is permitted to copy and distribute verbatim copies
 of this license document, but changing it is not allowed.

                            Preamble

  The licenses for most software are designed to take away your
freedom to share and change it.  By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users.  This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it.  (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.)  You can apply it to
your programs, too.

  When we speak of free software, we are referring to freedom, not
price.  Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.

  To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.

  For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have.  You must make sure that they, too, receive or can get the
source code.  And you must show them these terms so they know their
rights.

  We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.

  Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software.  If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.

  Finally, any free program is threatened constantly by software
patents.  We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary.  To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.

  The precise terms and conditions for copying, distribution and
modification follow.

                    GNU GENERAL PUBLIC LICENSE
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

  0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License.  The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language.  (Hereinafter, translation is included without limitation in
the term "modification".)  Each licensee is addressed as "you".

Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope.  The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.

  1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.

You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.

  2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:

    a) You must cause the modified files to carry prominent notices
    stating that you changed the files and the date of any change.

    b) You must cause any work that you distribute or publish, that in
    whole or in part contains or is derived from the Program or any
    part thereof, to be licensed as a whole at no charge to all third
    parties under the terms of this License.

    c) If the modified program normally reads commands interactively
    when run, you must cause it, when started running for such
    interactive use in the most ordinary way, to print or display an
    announcement including an appropriate copyright notice and a
    notice that there is no warranty (or else, saying that you provide
    a warranty) and that users may redistribute the program under
    these conditions, and telling the user how to view a copy of this
    License.  (Exception: if the Program itself is interactive but
    does not normally print such an announcement, your work based on
    the Program is not required to print an announcement.)

These requirements apply to the modified work as a whole.  If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works.  But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.

Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.

In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.

  3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:

    a) Accompany it with the complete corresponding machine-readable
    source code, which must be distributed under the terms of Sections
    1 and 2 above on a medium customarily used for software interchange; or,

    b) Accompany it with a written offer, valid for at least three
    years, to give any third party, for a charge no more than your
    cost of physically performing source distribution, a complete
    machine-readable copy of the corresponding source code, to be
    distributed under the terms of Sections 1 and 2 above on a medium
    customarily used for software interchange; or,

    c) Accompany it with the information you received as to the offer
    to distribute corresponding source code.  (This alternative is
    allowed only for noncommercial distribution and only if you
    received the program in object code or executable form with such
    an offer, in accord with Subsection b above.)

The source code for a work means the preferred form of the work for
making modifications to it.  For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable.  However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.

If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.

  4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License.  Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.

  5. You are not required to accept this License, since you have not
signed it.  However, nothing else grants you permission to modify or
distribute the Program or its derivative works.  These actions are
prohibited by law if you do not accept this License.  Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.

  6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions.  You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.

  7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License.  If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all.  For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.

If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.

It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices.  Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.

This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.

  8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded.  In such case, this License incorporates
the limitation as if written in the body of this License.

  9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time.  Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.

Each version is given a distinguishing version number.  If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation.  If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.

  10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission.  For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this.  Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.

                            NO WARRANTY

  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.

  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.

                     END OF TERMS AND CONDITIONS

            How to Apply These Terms to Your New Programs

  If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.

  To do so, attach the following notices to the program.  It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.

    {description}
    Copyright (C) {year}  {fullname}

    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation; either version 2 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License along
    with this program; if not, write to the Free Software Foundation, Inc.,
    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

Also add information on how to contact you by electronic and paper mail.

If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:

    Gnomovision version 69, Copyright (C) year name of author
    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
    This is free software, and you are welcome to redistribute it
    under certain conditions; type `show c' for details.

The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License.  Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.

You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary.  Here is a sample; alter the names:

  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
  `Gnomovision' (which makes passes at compilers) written by James Hacker.

  {signature of Ty Coon}, 1 April 1989
  Ty Coon, President of Vice

This General Public License does not permit incorporating your program into
proprietary programs.  If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library.  If this is what you want to do, use the GNU Lesser General
Public License instead of this License.
# cloudsync

**This project is obsolete and inactive. I recommend to use [rclone](https://rclone.org/) as a much more powerful alternative. If you are interested to takeover the ownership. Feel free to contact me.**

Sync a local filesystem on linux, windows and osx incremental and encrypted with google drive simliar to rsync. You can also restore the encrypted data back to a local filesystem. It works as a complete backup solution for your private data.

Other compareable backup solutions like [duplicity](http://duplicity.nongnu.org) upload one big encrypted 'base' archive with additional delta archive files. After a few months you must upload a new fresh 'base' archive to avoid hundreds of delta files. This approach, while widely used in various backup solutions, is problematic for private async DSL connections. To solve these issues each file is encrypted and uploaded separately.

To get a first impression you should take a look at [this screenshot](https://github.com/HolgerHees/cloudsync/wiki/Home).

The encryption is based on OpenPGP with AES 256 (optional) and a passphrase. It is possible to decrypt uploaded files with a normal OpenPGP compatible tool like 'gpg' or 'gpg2'.

Encrypted data includes:
- filetype [folder,file,symlink]
- filecontent, filename and original filesize
- createtime, modifytime and accesstime
- owner, group, posix permissions, acl entries and fat32 attributes
- md5 checksum

Filechanges are detected by comparing the file metadata. It uses a local cachefile to speedup the incremental update. The local cachefile is completly restoreable by analysing the serverside archived metadata.

Supported Services are:
- Google Drive (stable)
- Remote mounted filesystem (beta)
- Dropbox (alpha preview)

To provide additional cloud targets like Amazon Cloud Drive or Microsoft OneDrive just implement 6 functions from the interface [Connector.java](https://github.com/HolgerHees/cloudsync/tree/master/src/main/java/cloudsync/connector/RemoteConnector.java).

## Requirements

1. Java >= 7
2. [Java Cryptography Extension (JCE)](http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html) (optional to support AES256)
3. [Maven](http://maven.apache.org/download.cgi)

## Install

```bash
git clone https://github.com/HolgerHees/cloudsync.git
cd cloudsync
mvn install
```

## Config

To use, copy ```'config/cloudsync.config.default'``` to ```'config/cloudsync.config'``` and set the following;

- `PASSPHRASE`: your master password used for encrypt/decrypt

There are two authentication options, either using an Installed Application or Service Account.

Both options require that you [enable the Google Drive API](https://github.com/HolgerHees/cloudsync/wiki/Google-Drive-API). 

### Installed Application

Follow [these instructions](https://github.com/HolgerHees/cloudsync/wiki/Google-Client-Credentials) to set the following;

- `GOOGLE_DRIVE_CLIENT_ID`
- `GOOGLE_DRIVE_CLIENT_SECRET`

### Service Account

This process requires more configuration however will ensure you can run unattended.

1. [Create Service Account](https://developers.google.com/identity/protocols/OAuth2ServiceAccount). You must ensure that you [delegate domain-wide authority to your service account](https://developers.google.com/drive/web/delegation#delegate_domain-wide_authority_to_your_service_account).
2. Download P12 keyfile.
3. Set the following in the configuration file.

- `GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL` - ie.. `<some-id>@developer.gserviceaccount.com`.
- `GOOGLE_DRIVE_SERVICE_ACCOUNT_USER` - The Google Drive user, ie.. `you@yourdomain.com`.
- `GOOGLE_DRIVE_SERVICE_ACCOUNT_PRIVATE_KEY_P12_PATH` - The full or relative path to your P12 file.

## Usage

To create a backup of '/data', call:

```./cloudsync --backup /data --name dataBackup```

to restore a backup into '/restore', call:

```./cloudsync --restore /restore --name dataBackup```

for a complete list of options, see below:

```
usage: cloudsync <options>
 -b,--backup <path>                            Create or refresh backup of <path>
 -r,--restore <path>                           Restore a backup into <path>
 -c,--clean <path>                             Repair 'cloudsync*.cache' file and put leftover file into <path>
 -l,--list                                     List the contents of an backup
 -n,--name <name>                              Backup name of --backup, --restore, --clean or --list
    --config <path>                            Config file path. Default is './config/cloudsync.config'
    --followlinks <extern|all|none>            How to handle symbolic links
                                               <extern> - follow symbolic links if the target is outside from the
                                               current directory hierarchy - (default)
                                               <all> - follow all symbolic links
                                               <none> - don't follow any symbolic links
    --existing <stop|update|skip|rename>       Behavior on files that exists localy during --restore
                                               <stop> - stop immediately - (default)
                                               <update> - replace file
                                               <skip> - skip file
                                               <rename> - extend the name with an autoincrement number
    --history <count>                          Before remove or update a file or folder move it to a history folder.
                                               Use a maximum of <count> history folders
    --include <pattern>                        Include content of --backup, --restore and --list if the path matches the
                                               regex based ^<pattern>$. Multiple patterns can be separated with an '|'
                                               character.
    --exclude <pattern>                        Exclude content of --backup, --restore and --list if the path matches the
                                               regex based ^<pattern>$. Multiple patterns can be separated with an '|'
                                               character.
    --permissions <set|ignore|try>             Behavior how to handle acl permissions during --restore
                                               <set> - set all permissions and ownerships - (default)
                                               <ignore> - ignores all permissions and ownerships
                                               <try> - ignores invalid and not assignable permissions and ownerships
    --nocache                                  Don't use 'cloudsync*.cache' file for --backup or --list (much slower)
    --forcestart                               Ignore a existing pid file. Should only be used after a previous crashed
                                               job.
    --noencryption                             Don't encrypt uploaded data
    --dry-run                                  Perform a trial run of --backup or --restore with no changes made.
    --progress                                 Show progress during transfer and encryption.
    --retries <number>                         Number of network operation retries before an error is thrown (default:
                                               6).
    --waitretry <seconds>                      Number of seconds between 2 retries (default: 10).
    --network-error <exception|ask|continue>   How to continue on network problems
                                               <exception> - Throw an exception - (default)
                                               <ask> - Show a command prompt (Y/n) to continue
                                               <continue> - Show a warning and continue
    --file-error <exception|message>           How to continue on blocked files or permission problems
                                               <exception> - Throw an exception - (default)
                                               <message> - Show a error log message
    --logfile <path>                           Log message to <path>
    --cachefile <path>                         Cache data to <path>
    --min_tmp_file_size <size>                 Minimum file size <size> in bytes to use tmp files (default: 134217728)
 -v,--version                                  Show version number
 -h,--help                                     Show this help
 ```
@echo off

PATH %PATH%;%JAVA_HOME%\bin\
for /f tokens^=2-5^ delims^=.-_^" %%j in ('java -fullversion 2^>^&1') do set "jver=%%j%%k%%l%%m"

if  %jver% LSS 17000 (
    echo Found unsupported java version. Please install jdk 7 or newer
) else (
    java -cp ".\lib\*" cloudsync.Cloudsync %*
)

:end
#!/bin/bash

cd `dirname $0`

JAVA="java"

VERSION=$("$JAVA" -version 2>&1 | awk -F '"' '/version/ {print $2}')

if [[ "$VERSION" < 1.7 ]]; then
    echo Found unsupported java version. Please install jdk 7 or newer
else
    TIMESTAMP=`date +"%Y.%m.%d %H:%M:%S"`
    if [ -z "$LANG" ]; then
         export LANG="en_US.UTF-8"
         echo "[$TIMESTAMP] - WARNING - no \$LANG environment detected"
         echo "[$TIMESTAMP] - WARNING - set \"$LANG\" environment for local filename encoding"
    else
         echo "[$TIMESTAMP] - INFO - \"$LANG\" environment detected and used for local filename encoding"
    fi
    
    export LC_ALL=$LANG
    
    CMD="$JAVA -cp \"./lib/*\" cloudsync.Cloudsync $*"
    eval $CMD
fi
# your master password used for encrypt/decrypt
PASSPHRASE=

# use google drive as target
REMOTE_CONNECTOR=GoogleDrive

#
# Use either 'Installed Applications Auth' or 'Service Account Auth' sections below
#

# Installed Applications Auth - https://developers.google.com/api-client-library/java/google-api-java-client/oauth2#installed_applications
GOOGLE_DRIVE_CLIENT_ID=
GOOGLE_DRIVE_CLIENT_SECRET=
GOOGLE_DRIVE_CLIENT_TOKEN_PATH=./.cloudsync.token

# Service Account Auth - https://developers.google.com/api-client-library/java/google-api-java-client/oauth2#service_accounts
GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL=
GOOGLE_DRIVE_SERVICE_ACCOUNT_USER=
GOOGLE_DRIVE_SERVICE_ACCOUNT_PRIVATE_KEY_P12_PATH=

# backup target on the remote server
GOOGLE_DRIVE_DIR=/backup

### following options are also setable via command line. values from there are preferred ###
### {name} in *FILE properties is replaced with the currently used backup name ###

# used path to the cache file.
CACHEFILE=./.cloudsync_{name}.cache

# path to the log file
LOGFILE=./.cloudsync_{name}.log

# regex based exclude pattern
EXCLUDE=.*\\.git/.*|.*\\.svn/.*

# regex based include pattern
;INCLUDE=

# backup name
;NAME=

# behavior how to handle symbolic links
;FOLLOWLINKS=external

# behavior how to handle file duplications
;EXISTING=stop

# Maximum number of stored histories
;HISTORY=0

# create tmp file for encryption of files larger than (128MB)
# set a smaller value for low memory systems
;MIN_TMP_FILE_SIZE=134217728
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>org.cloudsync</groupId>
	<artifactId>cloudsync</artifactId>
	<version>v1.0-beta15</version>
	<name>cloudsync</name>
	<url>https://github.com/HolgerHees/cloudsync</url>
	<dependencies>
		<dependency>
			<groupId>commons-cli</groupId>
			<artifactId>commons-cli</artifactId>
			<version>1.3.1</version>
		</dependency>
		<dependency>
			<groupId>commons-codec</groupId>
			<artifactId>commons-codec</artifactId>
			<version>1.10</version>
		</dependency>
		<dependency>
			<groupId>org.apache.commons</groupId>
			<artifactId>commons-csv</artifactId>
			<version>1.4</version>
		</dependency>
		<dependency>
			<groupId>commons-io</groupId>
			<artifactId>commons-io</artifactId>
			<version>2.5</version>
		</dependency>
		<dependency>
			<groupId>org.apache.commons</groupId>
			<artifactId>commons-lang3</artifactId>
			<version>3.4</version>
		</dependency>
		<dependency>
			<groupId>org.bouncycastle</groupId>
			<artifactId>bcpg-jdk15on</artifactId>
			<version>1.54</version>
		</dependency>
		<dependency>
			<groupId>org.bouncycastle</groupId>
			<artifactId>bcprov-jdk15on</artifactId>
			<version>1.54</version>
		</dependency>
		<dependency>
			<groupId>com.google.apis</groupId>
			<artifactId>google-api-services-drive</artifactId>
			<version>v2-rev222-1.22.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.api-client</groupId>
			<artifactId>google-api-client</artifactId>
			<version>1.22.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.api-client</groupId>
			<artifactId>google-api-client-gson</artifactId>
			<version>1.22.0</version>
		</dependency>
		<dependency>
			<groupId>com.dropbox.core</groupId>
			<artifactId>dropbox-core-sdk</artifactId>
			<version>1.8</version>
		</dependency>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>4.10</version>
			<scope>test</scope>
		</dependency>
	</dependencies>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-clean-plugin</artifactId>
				<version>3.0.0</version>
				<configuration>
					<filesets>
						<fileset>
							<directory>lib</directory>
							<includes>
								<include>*.jar</include>
							</includes>
							<followSymlinks>false</followSymlinks>
						</fileset>
					</filesets>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>3.1</version>
				<configuration>
					<source>1.7</source>
					<target>1.7</target>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-jar-plugin</artifactId>
				<version>2.3</version>
				<configuration>
					<archive>
						<manifest>
							<mainClass>cloudsync.Cloudsync</mainClass>
							<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
							<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
						</manifest>
					</archive>
					<outputDirectory>${project.basedir}/lib</outputDirectory>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-assembly-plugin</artifactId>
				<version>2.4.1</version>
				<configuration>
					<descriptor>src/assembly/dep.xml</descriptor>
				</configuration>
				<executions>
					<execution>
						<phase>install</phase>
						<goals>
							<goal>single</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-dependency-plugin</artifactId>
				<version>2.8</version>
				<executions>
					<execution>
						<phase>install</phase>
						<goals>
							<goal>copy-dependencies</goal>
						</goals>
						<configuration>
							<outputDirectory>${project.basedir}/lib</outputDirectory>
						</configuration>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
</project>

<assembly
	xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
	<id>release</id>
	<formats>
		<format>tar.gz</format>
		<format>tar.bz2</format>
		<format>zip</format>
	</formats>
	<fileSets>
		<fileSet>
			<directory>${project.basedir}</directory>
			<outputDirectory>/</outputDirectory>
			<includes>
				<include>cloudsync.bat</include>
			</includes>
			<lineEnding>dos</lineEnding>
		</fileSet>
		<fileSet>
			<directory>${project.basedir}</directory>
			<outputDirectory>/</outputDirectory>
			<includes>
				<include>cloudsync.sh</include>
			</includes>
			<lineEnding>unix</lineEnding>
			<fileMode>0755</fileMode>
		</fileSet>
		<fileSet>
			<directory>${project.basedir}</directory>
			<outputDirectory>/</outputDirectory>
			<includes>
				<include>LICENSE</include>
				<include>README.md</include>
				<include>pom.xml</include>
				<include>config/cloudsync.config.default</include>
			</includes>
			<lineEnding>unix</lineEnding>
		</fileSet>
		<fileSet>
			<directory>${project.basedir}/src</directory>
			<useDefaultExcludes>true</useDefaultExcludes>
		</fileSet>
		<fileSet>
			<directory>${project.build.directory}</directory>
			<outputDirectory>/lib</outputDirectory>
			<includes>
				<include>*.jar</include>
			</includes>
		</fileSet>
	</fileSets>
	<dependencySets>
		<dependencySet>
			<outputDirectory>/lib</outputDirectory>
			<useProjectArtifact>true</useProjectArtifact>
			<scope>runtime</scope>
		</dependencySet>
	</dependencySets>
</assembly>
package cloudsync;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.util.logging.ConsoleHandler;
import java.util.logging.FileHandler;
import java.util.logging.Level;
import java.util.logging.Logger;

import cloudsync.exceptions.InfoException;
import org.apache.commons.cli.ParseException;
import org.apache.commons.lang3.StringUtils;

import cloudsync.connector.LocalFilesystemConnector;
import cloudsync.connector.RemoteConnector;
import cloudsync.exceptions.CloudsyncException;
import cloudsync.exceptions.UsageException;
import cloudsync.helper.CmdOptions;
import cloudsync.helper.Crypt;
import cloudsync.helper.Handler;
import cloudsync.logging.LogconsoleHandler;
import cloudsync.logging.LogfileFormatter;
import cloudsync.logging.LogfileHandler;
import cloudsync.model.options.SyncType;

public class Cloudsync
{
	private final static Logger	LOGGER	= Logger.getLogger(Cloudsync.class.getName());

	private final CmdOptions	options;

	public Cloudsync(final String[] args)
	{
		this.options = new CmdOptions(args);

		final Logger logger = Logger.getLogger("cloudsync");
		logger.setLevel(Level.ALL);

		java.util.logging.Handler[] handlers = logger.getHandlers();
		for(java.util.logging.Handler h : handlers) {
			h.close();
			logger.removeHandler(h);
		}
		final ConsoleHandler handler = new LogconsoleHandler();
		handler.setLevel(Level.ALL);
		logger.addHandler(handler);
		logger.setUseParentHandlers(false);
	}

	private void start() throws CloudsyncException, UsageException, InfoException
	{
		options.parse();

		String logpath = options.getLogfilePath();
		if (logpath != null)
		{
			final Logger logger = Logger.getLogger("cloudsync");
			FileHandler fh;
			try
			{
				Path logfilePath = Paths.get(logpath);
				if (Files.exists(logfilePath))
				{
					Path preservedPath = Paths.get(logpath + ".1");
					Files.move(logfilePath, preservedPath, StandardCopyOption.REPLACE_EXISTING);
				}
				fh = new LogfileHandler(logpath);
				fh.setFormatter(new LogfileFormatter());
				logger.addHandler(fh);
			}
			catch (SecurityException e)
			{
				throw new CloudsyncException("Unexpected error on logfile creation", e);
			}
			catch (IOException e)
			{
				throw new CloudsyncException("Unexpected error on logfile creation", e);
			}
		}

		String name = options.getName();

		final LocalFilesystemConnector localConnection = new LocalFilesystemConnector(options);

		Handler handler = null;

		try
		{
			String remoteConnectorName = options.getRemoteConnector();
			RemoteConnector remoteConnector = null;
			try
			{
				remoteConnector = (RemoteConnector) Class.forName("cloudsync.connector.Remote" + remoteConnectorName + "Connector").newInstance();
			}
			catch (IllegalAccessException e)
			{
			}
			catch (InstantiationException e)
			{
			}
			catch (ClassNotFoundException e)
			{
				throw new CloudsyncException("Remote connector '" + remoteConnectorName + "' not found", e);
			}

			remoteConnector.init(name, options);

			final long start = System.currentTimeMillis();

			SyncType type = options.getType();
			String[] includePatterns = options.getIncludePatterns();
			if (includePatterns != null)
			{
				LOGGER.log(Level.FINEST, "use include pattern: " + "[^" + StringUtils.join(includePatterns, "$] | [$") + "$]");
			}
			String[] excludePatterns = options.getExcludePatterns();
			if (excludePatterns != null)
			{
				LOGGER.log(Level.FINEST, "use exclude pattern: " + "[^" + StringUtils.join(excludePatterns, "$] | [$") + "$]");
			}

			handler = new Handler(name, localConnection, remoteConnector, options.getNoEncryption() ? null : new Crypt(options), options.getExistingBehavior(), options.getFollowLinks(),
					options.getPermissionType(),options.getFileErrorBehavior());
			handler.init(type, options.getCacheFile(), options.getLockFile(), options.getPIDFile(), options.getNoCache(), options.getForceStart());

			switch ( type )
			{
				case BACKUP:
					handler.backup(options.isDryRun(), includePatterns, excludePatterns);
					break;
				case RESTORE:
					handler.restore(options.isDryRun(), includePatterns, excludePatterns);
					break;
				case LIST:
					handler.list(includePatterns, excludePatterns);
					break;
				case CLEAN:
					handler.clean();
					break;
			}

			final long end = System.currentTimeMillis();

			LOGGER.log(Level.INFO, "runtime: " + ((end - start) / 1000.0f) + " seconds");
		}
		finally
		{
			if (handler != null) handler.finalize();
		}
	}

	public static void main(final String[] args) throws ParseException
	{
		final Cloudsync cloudsync = new Cloudsync(args);
		try
		{
			cloudsync.start();
		}
		catch (UsageException e)
		{
			if (!StringUtils.isEmpty(e.getMessage()))
			{
				LOGGER.log(Level.WARNING, e.getMessage() + "\n");
			}
			cloudsync.options.printHelp();
		}
		catch (InfoException e)
		{
			if (!StringUtils.isEmpty(e.getMessage()))
			{
				System.out.println("\n" + e.getMessage() + "\n");
			}
		}
		catch (CloudsyncException e)
		{
			LOGGER.log(Level.WARNING, e.getMessage() + "\n");
			if (e.getCause() != null)
			{
				e.printStackTrace();
			}
		}
	}
}
package cloudsync.connector;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.Charset;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.attribute.AclEntry;
import java.nio.file.attribute.AclEntry.Builder;
import java.nio.file.attribute.AclEntryFlag;
import java.nio.file.attribute.AclEntryPermission;
import java.nio.file.attribute.AclEntryType;
import java.nio.file.attribute.AclFileAttributeView;
import java.nio.file.attribute.BasicFileAttributeView;
import java.nio.file.attribute.BasicFileAttributes;
import java.nio.file.attribute.DosFileAttributeView;
import java.nio.file.attribute.DosFileAttributes;
import java.nio.file.attribute.FileOwnerAttributeView;
import java.nio.file.attribute.FileTime;
import java.nio.file.attribute.GroupPrincipal;
import java.nio.file.attribute.PosixFileAttributeView;
import java.nio.file.attribute.PosixFileAttributes;
import java.nio.file.attribute.PosixFilePermission;
import java.nio.file.attribute.UserPrincipal;
import java.nio.file.attribute.UserPrincipalLookupService;
import java.nio.file.attribute.UserPrincipalNotFoundException;
import java.text.DecimalFormat;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.logging.Level;
import java.util.logging.Logger;

import cloudsync.exceptions.FileIOException;
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.io.FilenameUtils;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.helper.CmdOptions;
import cloudsync.helper.Handler;
import cloudsync.helper.Helper;
import cloudsync.model.options.ExistingType;
import cloudsync.model.Item;
import cloudsync.model.ItemType;
import cloudsync.model.options.FollowLinkType;
import cloudsync.model.options.PermissionType;
import cloudsync.model.LocalStreamData;
import cloudsync.model.RemoteStreamData;

public class LocalFilesystemConnector
{
	private static final Logger							LOGGER			= Logger.getLogger(LocalFilesystemConnector.class.getName());

	private static final int							BUFFER_SIZE		= 1 << 16;
	private static final DecimalFormat					df				= new DecimalFormat("00");

	private static final Map<Integer, PosixFilePermission>	toPermMapping	= new HashMap<>();
	static
	{
		toPermMapping.put(0001, PosixFilePermission.OTHERS_EXECUTE);
		toPermMapping.put(0002, PosixFilePermission.OTHERS_WRITE);
		toPermMapping.put(0004, PosixFilePermission.OTHERS_READ);
		toPermMapping.put(0010, PosixFilePermission.GROUP_EXECUTE);
		toPermMapping.put(0020, PosixFilePermission.GROUP_WRITE);
		toPermMapping.put(0040, PosixFilePermission.GROUP_READ);
		toPermMapping.put(0100, PosixFilePermission.OWNER_EXECUTE);
		toPermMapping.put(0200, PosixFilePermission.OWNER_WRITE);
		toPermMapping.put(0400, PosixFilePermission.OWNER_READ);
	}

	private static final Map<PosixFilePermission, Integer>	fromPermMapping	= new HashMap<>();

	private static final Map<String, Boolean>			principal_state	= new HashMap<>();

	private final String								localPath;
	private final boolean								showProgress;

	public LocalFilesystemConnector(final CmdOptions options)
	{
		String path = options.getPath();
		showProgress = options.showProgress();

		if (path != null)
		{
			if (path.startsWith(Item.SEPARATOR))
			{
				localPath = Item.SEPARATOR + Helper.trim(path, Item.SEPARATOR);
			}
			else
			{
				localPath = Helper.trim(path, Item.SEPARATOR);
			}
		}
		else
		{
			localPath = "";
		}

		for (final Integer key : toPermMapping.keySet())
		{
			final PosixFilePermission perm = toPermMapping.get(key);
			fromPermMapping.put(perm, key);
		}
	}

	public void prepareUpload(final Handler handler, final Item item, final ExistingType duplicateFlag)
	{
		if (!duplicateFlag.equals(ExistingType.RENAME))
		{
			return;
		}

		String path = localPath + Item.SEPARATOR + item.getPath();

		if (exists(Paths.get(path)))
		{
			int i = 0;
			while (exists(Paths.get(path + "." + i)))
			{
				i++;
			}

			path += "." + i;

			item.setName(FilenameUtils.getName(path));
		}
	}

	public void prepareParent(Handler handler, Item item) throws CloudsyncException
	{
		if (item.getParent() != null)
		{
			Item parentItem = item.getParent();

			final Path parentPath = Paths.get(localPath + Item.SEPARATOR + parentItem.getPath());

			try
			{
				Files.createDirectories(parentPath);
			}
			catch (IOException e)
			{
				throw new CloudsyncException("Can't create " + parentItem.getTypeName() + " '" + parentItem.getPath() + "'", e);
			}
		}
	}

	public void upload(final Handler handler, final Item item, final ExistingType duplicateFlag, final PermissionType permissionType)
			throws CloudsyncException
	{
		final String _path = localPath + Item.SEPARATOR + item.getPath();

		final Path path = Paths.get(_path);

		if (exists(path))
		{
			if (duplicateFlag.equals(ExistingType.SKIP))
			{
				return;
			}

			if (!duplicateFlag.equals(ExistingType.UPDATE))
			{
				throw new CloudsyncException("Item '" + item.getPath() + "' already exists. Try to specify another '--duplicate' behavior.");
			}

			if ((!item.isType(ItemType.FOLDER) || !isDir(path)))
			{
				try
				{
					Files.delete(path);
				}
				catch (final IOException e)
				{
					throw new CloudsyncException("Can't clear " + item.getTypeName() + " on '" + item.getPath() + "'", e);
				}
			}
		}

		if (item.isType(ItemType.FOLDER))
		{
			if (!exists(path))
			{
				try
				{
					Files.createDirectory(path);
				}
				catch (final IOException e)
				{
					throw new CloudsyncException("Can't create " + item.getTypeName() + " '" + item.getPath() + "'", e);
				}
			}
		}
		else
		{
			if (item.getParent() != null)
			{
				final Path parentPath = Paths.get(localPath + Item.SEPARATOR + item.getParent().getPath());

				if (!isDir(parentPath))
				{

					throw new CloudsyncException("Parent directory of " + item.getTypeName() + " '" + item.getPath() + "' is missing.");
				}
			}

			if (item.isType(ItemType.LINK))
			{
				RemoteStreamData remoteStreamData = null;

				try
				{
					remoteStreamData = handler.getRemoteProcessedBinary(item);

					final String link = IOUtils.toString( remoteStreamData.getDecryptedStream(), Charset.defaultCharset());
					Files.createSymbolicLink(path, Paths.get(link));

				}
				catch (final IOException e)
				{
					throw new CloudsyncException("Unexpected error during local update of " + item.getTypeName() + " '" + item.getPath() + "'", e);

				}
				finally
				{
					if (remoteStreamData != null) remoteStreamData.close();
				}
			}
			else if (item.isType(ItemType.FILE))
			{
				RemoteStreamData remoteStreamData = null;
				OutputStream outputStream = null;
				InputStream localChecksumStream = null;

				try
				{
					remoteStreamData = handler.getRemoteProcessedBinary(item);
					outputStream = Files.newOutputStream(path);

					final long length = item.getFilesize();
					double current = 0;

					byte[] buffer = new byte[BUFFER_SIZE];
					int len;

					// 2 MB
					if (showProgress && length > 2097152)
					{

						long lastTime = System.currentTimeMillis();
						double lastBytes = 0;
						String currentSpeed = "";

						while ((len = remoteStreamData.getDecryptedStream().read(buffer)) != -1)
						{
							outputStream.write(buffer, 0, len);
							current += len;

							long currentTime = System.currentTimeMillis();

							String msg = "\r  " + df.format(Math.ceil(current * 100 / length)) + "% (" + convertToKB(current) + " of " + convertToKB(length)
									+ " kb) restored";
							double diffTime = ((currentTime - lastTime) / 1000.0);

							if (diffTime > 5.0)
							{
								long speed = convertToKB((current - lastBytes) / diffTime);
								currentSpeed = " - " + speed + " kb/s";

								lastTime = currentTime;
								lastBytes = current;
							}

							LOGGER.log(Level.FINEST, msg + currentSpeed, true);
						}
					}
					else
					{
						while ((len = remoteStreamData.getDecryptedStream().read(buffer)) != -1)
						{
							outputStream.write(buffer, 0, len);
						}
					}

					localChecksumStream = Files.newInputStream(path);

					if (!createChecksum(localChecksumStream).equals(item.getChecksum()))
					{
						throw new CloudsyncException("restored filechecksum differs from the original filechecksum");
					}
					if (item.getFilesize() != Files.size(path))
					{
						throw new CloudsyncException("restored filesize differs from the original filesize");
					}

				}
				catch (final IOException e)
				{
					throw new CloudsyncException("Unexpected error during local update of " + item.getTypeName() + " '" + item.getPath() + "'", e);

				}
				finally
				{
					if (remoteStreamData != null) remoteStreamData.close();
					if (outputStream != null) IOUtils.closeQuietly(outputStream);
					if (localChecksumStream != null) IOUtils.closeQuietly(localChecksumStream);
				}

			}
			else
			{
				throw new CloudsyncException("Unsupported type " + item.getTypeName() + "' on '" + item.getPath() + "'");
			}
		}

		try
		{
			if (item.isType(ItemType.LINK))
			{
				// Files.setLastModifiedTime(path, item.getModifyTime());
			}
			else
			{
				Files.getFileAttributeView(path, BasicFileAttributeView.class, LinkOption.NOFOLLOW_LINKS).setTimes(item.getModifyTime(), item.getAccessTime(),
						item.getCreationTime());
			}
		}
		catch (final IOException e)
		{
			throw new CloudsyncException("Can't set create, modify and access time of " + item.getTypeName() + " '" + item.getPath() + "'", e);
		}

		if (permissionType.equals(PermissionType.SET) || permissionType.equals(PermissionType.TRY))
		{
			final UserPrincipalLookupService lookupService = FileSystems.getDefault().getUserPrincipalLookupService();

			Map<String, String[]> attributes = item.getAttributes();
			for (String type : attributes.keySet())
			{
				GroupPrincipal group;
				UserPrincipal principal;

				try
				{
					String[] values = attributes.get(type);

					switch ( type )
					{
						case Item.ATTRIBUTE_POSIX:
							PosixFileAttributeView posixView = Files.getFileAttributeView(path, PosixFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
							if (posixView != null)
							{
								group = lookupService.lookupPrincipalByGroupName(values[0]);
								posixView.setGroup(group);
								principal = lookupService.lookupPrincipalByName(values[1]);
								posixView.setOwner(principal);
								if (values.length > 2) posixView.setPermissions(toPermissions(Integer.parseInt(values[2])));
							}
							else
							{
								String msg = "Can't restore 'posix' permissions on '" + item.getPath() + "'. They are not supported.";
								if (permissionType.equals(PermissionType.TRY)) LOGGER.log(Level.WARNING, msg);
								else throw new CloudsyncException(msg + "\n  try to run with '--permissions try'");
							}
							break;
						case Item.ATTRIBUTE_DOS:
							DosFileAttributeView dosView = Files.getFileAttributeView(path, DosFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
							if (dosView != null)
							{
								dosView.setArchive(Boolean.parseBoolean(values[0]));
								dosView.setHidden(Boolean.parseBoolean(values[1]));
								dosView.setReadOnly(Boolean.parseBoolean(values[2]));
								dosView.setSystem(Boolean.parseBoolean(values[3]));
							}
							else
							{
								String msg = "Can't restore 'dos' permissions on '" + item.getPath() + "'. They are not supported.";
								if (permissionType.equals(PermissionType.TRY)) LOGGER.log(Level.WARNING, msg);
								else throw new CloudsyncException(msg + "\n  try to run with '--permissions try'");
							}
							break;
						case Item.ATTRIBUTE_ACL:
							AclFileAttributeView aclView = Files.getFileAttributeView(path, AclFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
							if (aclView != null)
							{
								List<AclEntry> acls = aclView.getAcl();
								for (int i = 0; i < values.length; i = i + 4)
								{
									Builder aclEntryBuilder = AclEntry.newBuilder();

									aclEntryBuilder.setType(AclEntryType.valueOf(values[i]));
									aclEntryBuilder.setPrincipal(lookupService.lookupPrincipalByName(values[i + 1]));

									Set<AclEntryFlag> flags = new HashSet<>();
									for (String flag : StringUtils.splitPreserveAllTokens(values[i + 2], ","))
									{
										flags.add(AclEntryFlag.valueOf(flag));
									}
									if (flags.size() > 0) aclEntryBuilder.setFlags(flags);

									Set<AclEntryPermission> aclPermissions = new HashSet<>();
									for (String flag : StringUtils.splitPreserveAllTokens(values[i + 3], ","))
									{
										aclPermissions.add(AclEntryPermission.valueOf(flag));
									}
									if (aclPermissions.size() > 0) aclEntryBuilder.setPermissions(aclPermissions);
									acls.add(aclEntryBuilder.build());
								}
								aclView.setAcl(acls);
							}
							else
							{
								String msg = "Can't restore 'acl' permissions on '" + item.getPath() + "'. They are not supported.";
								if (permissionType.equals(PermissionType.TRY)) LOGGER.log(Level.WARNING, msg);
								else throw new CloudsyncException(msg + "\n  try to run with '--permissions try'");
							}
							break;
						case Item.ATTRIBUTE_OWNER:
							FileOwnerAttributeView ownerView = Files.getFileAttributeView(path, FileOwnerAttributeView.class, LinkOption.NOFOLLOW_LINKS);
							if (ownerView != null)
							{
								principal = lookupService.lookupPrincipalByName(values[0]);
								ownerView.setOwner(principal);
							}
							else
							{
								String msg = "Can't restore 'owner' permissions on '" + item.getPath() + "'. They are not supported.";
								if (permissionType.equals(PermissionType.TRY)) LOGGER.log(Level.WARNING, msg);
								else throw new CloudsyncException(msg + "\n  try to run with '--permissions try'");
							}
							break;
					}
				}
				catch (final UserPrincipalNotFoundException e)
				{
					if (!LocalFilesystemConnector.principal_state.containsKey(e.getName()))
					{
						LocalFilesystemConnector.principal_state.put(e.getName(), true);
						LOGGER.log(Level.WARNING, "principal with name '" + e.getName() + "' not exists");
					}
					String msg = "Principal '" + e.getName() + "' on '" + item.getPath() + "' not found.";
					if (permissionType.equals(PermissionType.TRY)) LOGGER.log(Level.WARNING, msg);
					else throw new CloudsyncException(msg + "\n  try to run with '--permissions try'");
				}
				catch (final IOException e)
				{
					String msg = "Can't set permissions of '" + item.getPath() + "'.";
					if (permissionType.equals(PermissionType.TRY)) LOGGER.log(Level.WARNING, msg);
					else throw new CloudsyncException(msg + "\n  try to run with '--permissions try'");
				}
			}
		}
	}

	public File[] readFolder(final Item item) 
	
	throws CloudsyncException
	{
		final String currentPath = localPath + (StringUtils.isEmpty(item.getPath()) ? "" : Item.SEPARATOR + item.getPath());

		final File folder = new File(currentPath);

		if (!Files.exists(folder.toPath(), LinkOption.NOFOLLOW_LINKS))
		{
			LOGGER.log(Level.WARNING, "skip '" + currentPath + "'. does not exists anymore.");
			return new File[] {};
		}

		File[] files = folder.listFiles();
		
		if( files == null )
		{
            throw new CloudsyncException("Path '" + currentPath + "' is not readable. Check your permissions");
		}
		
		return files;
	}

	public Item getItem(File file, final FollowLinkType followlinks, List<String> followedLinkPaths ) throws FileIOException
	{
		try
		{
			Path path = file.toPath();

			ItemType type;
			
			if (Files.isSymbolicLink(path))
			{
				String target;
				target = Files.readSymbolicLink(path).toString();
				final String firstChar = target.substring(0, 1);
				if (!firstChar.equals(Item.SEPARATOR))
				{
					if (!firstChar.equals("."))
					{
						target = "." + Item.SEPARATOR + target;
					}
					target = path.toString() + Item.SEPARATOR + target;
				}
				target = Paths.get(target).toFile().getCanonicalPath();

                if (!followlinks.equals(FollowLinkType.NONE) && followlinks.equals(FollowLinkType.EXTERNAL) && !Paths.get(target).toFile().getCanonicalFile().toPath().startsWith(localPath + Item.SEPARATOR) )
				{
                    boolean foundLink = false;
                    for( String followedLinkPath: followedLinkPaths )
                    {
                        // 1. if the link target is a child of a already followed link, then there is no need to follow again
                        // 2. and the target should not be equal with a already followed link. Otherwise we are requesting the same item again. So we have to follow.
                        if( target.startsWith(followedLinkPath) && !target.equals(followedLinkPath) )
                        {
                            foundLink = true;
                            break;
                        }
                    }
                    
                    if( !foundLink )
                    {
                        final Path targetPath = Paths.get(target);
                        if (Files.exists(targetPath, LinkOption.NOFOLLOW_LINKS))
                        {
                            path = targetPath;
                            followedLinkPaths.add(target);
                        }
                    }
				}
			}

			BasicFileAttributes basic_attr = Files.readAttributes(path, BasicFileAttributes.class, LinkOption.NOFOLLOW_LINKS);
			final Long filesize = basic_attr.size();
			final FileTime creationTime = basic_attr.creationTime();
			final FileTime modifyTime = basic_attr.lastModifiedTime();
			final FileTime accessTime = basic_attr.lastAccessTime();

			if (basic_attr.isDirectory())
			{
				type = ItemType.FOLDER;
			}
			else if (basic_attr.isRegularFile())
			{
				type = ItemType.FILE;
			}
			else if (basic_attr.isSymbolicLink())
			{
				type = ItemType.LINK;
			}
			else
			{
				type = ItemType.UNKNOWN;
			}

			Map<String, String[]> attributes = new HashMap<>();

			PosixFileAttributeView posixView = Files.getFileAttributeView(path, PosixFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
			if (posixView != null)
			{
				final PosixFileAttributes attr = posixView.readAttributes();
				if (type.equals(ItemType.LINK))
				{
					attributes.put(Item.ATTRIBUTE_POSIX, new String[] { attr.group().getName(), attr.owner().getName() });
				}
				else
				{
					attributes.put(Item.ATTRIBUTE_POSIX, new String[] { attr.group().getName(), attr.owner().getName(),
							fromPermissions(attr.permissions()).toString() });
				}
			}
			else
			{
				DosFileAttributeView dosView = Files.getFileAttributeView(path, DosFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
				if (dosView != null)
				{
					final DosFileAttributes attr = dosView.readAttributes();
					attributes.put(Item.ATTRIBUTE_DOS, new String[] { attr.isArchive() ? "1" : "0", attr.isHidden() ? "1" : "0", attr.isReadOnly() ? "1" : "0",
							attr.isSystem() ? "1" : "0" });
				}
			}

			if (!type.equals(ItemType.LINK))
			{
				AclFileAttributeView aclView = Files.getFileAttributeView(path, AclFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
				if (aclView != null)
				{
					if (!attributes.containsKey(Item.ATTRIBUTE_POSIX)) attributes.put(Item.ATTRIBUTE_OWNER, new String[] { aclView.getOwner().getName() });

					AclFileAttributeView parentAclView = Files.getFileAttributeView(path.getParent(), AclFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);

					List<AclEntry> aclList = getLocalAclEntries(type, parentAclView.getAcl(), aclView.getAcl());
					if (aclList.size() > 0)
					{
						List<String> aclData = new ArrayList<>();
						for (AclEntry acl : aclList)
						{
							List<String> flags = new ArrayList<>();
							for (AclEntryFlag flag : acl.flags())
							{
								flags.add(flag.name());
							}
							List<String> permissions = new ArrayList<>();
							for (AclEntryPermission permission : acl.permissions())
							{
								permissions.add(permission.name());
							}

							aclData.add(acl.type().name());
							aclData.add(acl.principal().getName());
							aclData.add(StringUtils.join(flags, ","));
							aclData.add(StringUtils.join(permissions, ","));
						}
						String[] arr = new String[aclData.size()];
						arr = aclData.toArray(arr);
						attributes.put(Item.ATTRIBUTE_ACL, arr);
					}
				}
				else if (!attributes.containsKey(Item.ATTRIBUTE_POSIX))
				{
					FileOwnerAttributeView ownerView = Files.getFileAttributeView(path, FileOwnerAttributeView.class, LinkOption.NOFOLLOW_LINKS);
					if (ownerView != null)
					{
						attributes.put(Item.ATTRIBUTE_OWNER, new String[] { ownerView.getOwner().getName() });
					}
				}
			}

			return Item.fromLocalData(file.getName(), type, filesize, creationTime, modifyTime, accessTime, attributes);
		}
		catch (final IOException e)
		{
			throw new FileIOException("Can't read attributes of '" + file.getAbsolutePath() + "'", e);
		}
	}

	private List<AclEntry> getLocalAclEntries(ItemType type, List<AclEntry> parentAclList, List<AclEntry> childAclList)
	{
		List<AclEntry> aclList = new ArrayList<>();

		for (AclEntry childEntry : childAclList)
		{
			boolean found = false;
			for (AclEntry parentEntry : parentAclList)
			{
				if (!parentEntry.type().equals(childEntry.type())) continue;
				if (!parentEntry.principal().equals(childEntry.principal())) continue;
				if (!parentEntry.permissions().equals(childEntry.permissions())) continue;
				if (!parentEntry.flags().equals(childEntry.flags()))
				{
					if (parentEntry.flags().contains(AclEntryFlag.INHERIT_ONLY))
					{
						found = true;
						break;
					}
					else
					{
						if (type.equals(ItemType.FOLDER))
						{
							if (parentEntry.flags().contains(AclEntryFlag.DIRECTORY_INHERIT))
							{
								found = true;
								break;
							}
						}
						else
						{
							if (parentEntry.flags().contains(AclEntryFlag.FILE_INHERIT))
							{
								found = true;
								break;
							}
						}
					}
					continue;
				}
				found = true;
				break;
			}

			if (found) continue;

			// System.out.println("CHILD: "+childEntry.toString());
			/*
			 * System.out.println("\n\n");
			 * System.out.println("CHILD: "+childEntry.toString());
			 * 
			 * for(AclEntry parentEntry : parentAclList){
			 * 
			 * System.out.println("PARENT: "+parentEntry.toString()); }
			 * 
			 * System.out.println("\n\n");
			 */

			aclList.add(childEntry);
		}
		return aclList;
	}

	private boolean exists(final Path path)
	{
		return Files.exists(path, LinkOption.NOFOLLOW_LINKS);
	}

	private boolean isDir(final Path path)
	{
		return Files.isDirectory(path, LinkOption.NOFOLLOW_LINKS);
	}

	private Integer fromPermissions(final Set<PosixFilePermission> posixPerms)
	{
		int result = 0;
		for (final PosixFilePermission posixPerm : posixPerms)
		{
			result += fromPermMapping.get(posixPerm);
		}
		return result;
	}

	private Set<PosixFilePermission> toPermissions(final Integer perm)
	{
		final int mode = perm;
		final Set<PosixFilePermission> permissions = new HashSet<>();
		for (final int mask : toPermMapping.keySet())
		{
			if (mask == (mode & mask))
			{
				permissions.add(toPermMapping.get(mask));
			}
		}
		return permissions;
	}

	private static String createChecksum(final InputStream data) throws IOException
	{
		return DigestUtils.md5Hex(data);
	}

	public LocalStreamData getFileBinary(final Item item) throws FileIOException
	{
		File file = new File(localPath + Item.SEPARATOR + item.getPath());

		InputStream checksumInputStream = null;

		try
		{
			if (item.isType(ItemType.LINK))
			{
				byte[] data = Files.readSymbolicLink(file.toPath()).toString().getBytes();
				checksumInputStream = new ByteArrayInputStream(data);
				item.setChecksum(createChecksum(checksumInputStream));

				return new LocalStreamData(new ByteArrayInputStream(data), data.length);

			}
			else if (item.isType(ItemType.FILE))
			{
				checksumInputStream = Files.newInputStream(file.toPath());
				item.setChecksum(createChecksum(checksumInputStream));

				return new LocalStreamData(Files.newInputStream(file.toPath()), Files.size(file.toPath()));
			}
			return null;

		}
		catch (final IOException e)
		{
			throw new FileIOException("Can't read data of '" + file.getAbsolutePath() + "'", e);

		}
		finally
		{
			if (checksumInputStream != null) IOUtils.closeQuietly(checksumInputStream);
		}
	}

	private long convertToKB(double size)
	{
		return (long) Math.ceil(size / 1024);
	}
}
package cloudsync.connector;

import java.io.InputStream;
import java.util.List;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.exceptions.FileIOException;
import cloudsync.helper.CmdOptions;
import cloudsync.helper.Handler;
import cloudsync.model.Item;
import cloudsync.model.RemoteItem;

public interface RemoteConnector
{
	void init(String backupName, CmdOptions options) throws CloudsyncException;

	List<RemoteItem> readFolder(Handler handler, Item parentItem) throws CloudsyncException;

	void upload(Handler handler, Item item) throws CloudsyncException, FileIOException;

	void update(Handler handler, Item item, boolean with_filedata) throws CloudsyncException, FileIOException;

	void remove(Handler handler, Item item) throws CloudsyncException;

	InputStream get(Handler handler, Item item) throws CloudsyncException;

	void cleanHistory(Handler handler) throws CloudsyncException;
}
package cloudsync.connector;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.attribute.FileTime;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;

import cloudsync.exceptions.FileIOException;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.StringUtils;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.helper.CmdOptions;
import cloudsync.helper.Handler;
import cloudsync.helper.Helper;
import cloudsync.model.Item;
import cloudsync.model.ItemType;
import cloudsync.model.RemoteItem;
import cloudsync.model.LocalStreamData;

import com.dropbox.core.DbxAppInfo;
import com.dropbox.core.DbxAuthFinish;
import com.dropbox.core.DbxClient;
import com.dropbox.core.DbxEntry;
import com.dropbox.core.DbxEntry.File;
import com.dropbox.core.DbxException;
import com.dropbox.core.DbxRequestConfig;
import com.dropbox.core.DbxWebAuthNoRedirect;
import com.dropbox.core.DbxWriteMode;

public class RemoteDropboxConnector implements RemoteConnector
{
	private final static Logger		LOGGER				= Logger.getLogger(RemoteDropboxConnector.class.getName());

	private final static String		SEPARATOR			= "/";
	private final static String		METADATA_SUFFIX		= ".md";

	private final static int		MIN_SEARCH_BREAK	= 5000;
	private final static int		MIN_SEARCH_RETRIES	= 12;
	private final static int		MIN_RETRY_BREAK		= 10000;
	private final static int		RETRY_COUNT			= 6;														// =>

	private String					backupRootPath;
	private String					backupHistoryPath;

	private Map<String, DbxEntry>	cacheFiles;

	private Path					tokenPath;
	private String					basePath;
	private String					backupName;
	private Integer					historyCount;

	private long					lastValidate		= 0;

	private DbxClient				client;

	private boolean					isInitialized;

	private Charset                 charset;

	public RemoteDropboxConnector()
	{
	}

	@Override
	public void init(String backupName, CmdOptions options) throws CloudsyncException
	{
		RemoteDropboxOptions dropboxOptions = new RemoteDropboxOptions(options, backupName);
		Integer history = options.getHistory();

		cacheFiles = new HashMap<>();

		this.basePath = Helper.trim(dropboxOptions.getBasePath(), SEPARATOR);
		this.backupName = backupName;
		this.historyCount = history;
		this.tokenPath = Paths.get(dropboxOptions.getTokenPath());

		this.backupRootPath = basePath + SEPARATOR + backupName;
		this.backupHistoryPath = history > 0 ? this.backupRootPath + " " + new SimpleDateFormat("yyyy.MM.dd_HH.mm.ss").format(new Date()) : null;

		try
		{
			String token = Files.exists(this.tokenPath) ? FileUtils.readFileToString(this.tokenPath.toFile(), charset) : null;

			DbxAppInfo appInfo = new DbxAppInfo(dropboxOptions.getAppKey(), dropboxOptions.getAppSecret());

			DbxRequestConfig config = new DbxRequestConfig("Cloudsync/1.0", Locale.getDefault().toString());
			DbxWebAuthNoRedirect webAuth = new DbxWebAuthNoRedirect(config, appInfo);

			if (StringUtils.isEmpty(token))
			{
				final String url = webAuth.start();
				System.out
						.println("Please open the following URL in your browser, click \"Allow\" (you might have to log in first) and copy the authorization code and enter below");
				System.out.println("\n" + url + "\n");
				final String code = new BufferedReader(new InputStreamReader(System.in)).readLine().trim();

				DbxAuthFinish authFinish = webAuth.finish(code);
				token = authFinish.accessToken;

				FileUtils.write(this.tokenPath.toFile(), token, charset);

				LOGGER.log(Level.INFO, "client token stored in '" + this.tokenPath + "'");
			}

			client = new DbxClient(config, token);

		}
		catch (DbxException e)
		{
			throw new CloudsyncException("Can't init remote dropbox connector", e);
		}
		catch (IOException e)
		{
			throw new CloudsyncException("Can't init remote dropbox connector", e);
		}
	}

	@Override
	public void upload(final Handler handler, final Item item) throws CloudsyncException, FileIOException
	{
		initService(handler);

		int retryCount = 0;
		do
		{
			try
			{
				Item parentItem = item.getParent();
				String parentPath = buildPath(parentItem);
				String path = parentPath + SEPARATOR + handler.getLocalProcessedTitle(item);
				DbxEntry entry;

				if (item.isType(ItemType.FOLDER))
				{
					entry = client.createFolder(path);
				}
				else
				{
					LocalStreamData data = handler.getLocalProcessedBinary(item);
					if (data == null) data = new LocalStreamData(new ByteArrayInputStream("".getBytes()), 0);
					entry = client.uploadFile(path, DbxWriteMode.add(), data.getLength(), data.getStream());
				}

				String metadata = handler.getLocalProcessedMetadata(item);

				client.uploadFile(path + METADATA_SUFFIX, DbxWriteMode.add(), metadata.length(), new ByteArrayInputStream(metadata.getBytes("ASCII")));
				_addToCache(entry);
				item.setRemoteIdentifier(entry.name);
				return;
			}
			catch (final IOException e)
			{
				retryCount = validateException("remote upload", item, e, retryCount);
				// TODO search for interrupted uploads
			}
			catch (final DbxException e)
			{
				retryCount = validateException("remote upload", item, e, retryCount);
				// TODO search for interrupted uploads
			}
		}
		while (true);
	}

	@Override
	public void update(final Handler handler, final Item item, final boolean with_filedata) throws CloudsyncException, FileIOException
	{
		initService(handler);

		int retryCount = 0;
		do
		{
			try
			{
				String path = buildPath(item);
				if (with_filedata)
				{
					LocalStreamData data = handler.getLocalProcessedBinary(item);
					if (data != null)
					{
						client.uploadFile(path, DbxWriteMode.force(), data.getLength(), data.getStream());
					}
				}
				String metadata = handler.getLocalProcessedMetadata(item);
				client.uploadFile(path + METADATA_SUFFIX, DbxWriteMode.force(), metadata.length(), new ByteArrayInputStream(metadata.getBytes("ASCII")));
				return;
			}
			catch (final IOException e)
			{
				retryCount = validateException("remote update", item, e, retryCount);
			}
			catch (final DbxException e)
			{
				retryCount = validateException("remote update", item, e, retryCount);
			}
		}
		while (true);
	}

	@Override
	public void remove(final Handler handler, final Item item) throws CloudsyncException
	{
		initService(handler);

		int retryCount = 0;
		do
		{
			try
			{
				String path = buildPath(item);
				if (backupHistoryPath != null)
				{
					client.move(path, path.replaceFirst(backupRootPath, backupHistoryPath));
					client.move(path, path.replaceFirst(backupRootPath + METADATA_SUFFIX, backupHistoryPath + METADATA_SUFFIX));
				}
				else
				{
					client.delete(path);
				}
				_removeFromCache(path);
				return;
			}
			catch (final DbxException e)
			{
				retryCount = validateException("remote remove", item, e, retryCount);
			}
		}
		while (true);
	}

	@Override
	public InputStream get(final Handler handler, final Item item) throws CloudsyncException
	{
		initService(handler);

		int retryCount = 0;
		do
		{
			try
			{
				try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream())
				{
					client.getFile(buildPath(item), null, outputStream);
					return new ByteArrayInputStream(outputStream.toByteArray());
				}
			}
			catch (final DbxException e)
			{
				retryCount = validateException("fetch metadata", null, e, retryCount);
			}
			catch (final IOException e)
			{
				retryCount = validateException("fetch metadata", null, e, retryCount);
			}
		}
		while (true);
	}

	@Override
	public List<RemoteItem> readFolder(final Handler handler, final Item parentItem) throws CloudsyncException
	{
		initService(handler);

		int retryCount = 0;
		do
		{
			try
			{
				// refreshCredential();

				final List<RemoteItem> child_items = new ArrayList<>();
				Map<String, DbxEntry[]> childContainer = new HashMap<>();
				DbxEntry.WithChildren listing = client.getMetadataWithChildren(buildPath(parentItem));
				for (DbxEntry child : listing.children)
				{
					String[] nameParts = child.name.split("\\.");

					DbxEntry[] entries = childContainer.get(nameParts[0]);
					if (entries == null) entries = new DbxEntry[2];
					if (nameParts.length == 2) entries[1] = child;
					else entries[0] = child;

					childContainer.put(nameParts[0], entries);

				}

				for (final DbxEntry[] childData : childContainer.values())
				{
					child_items.add(_prepareBackupItem(childData, handler));
				}
				return child_items;
			}
			catch (final DbxException e)
			{
				retryCount = validateException("remote fetch", parentItem, e, retryCount);
			}
		}
		while (true);
	}

	@Override
	public void cleanHistory(final Handler handler) throws CloudsyncException
	{
		initService(handler);
		try
		{
			final List<DbxEntry> child_items = new ArrayList<>();
			for (DbxEntry entry : client.getMetadataWithChildren(basePath).children)
			{
				if (!entry.name.startsWith(backupName) || entry.name.equals(backupName)) continue;

				child_items.add(entry);
			}

			if (child_items.size() > historyCount)
			{
				Collections.sort(child_items, new Comparator<DbxEntry>()
				{
					@Override
					public int compare(final DbxEntry o1, final DbxEntry o2)
					{
						return o1.name.compareTo(o2.name);
					}
				});

				for (DbxEntry entry : child_items.subList(historyCount, child_items.size()))
				{
					LOGGER.log(Level.FINE, "cleanup history folder '" + entry.name + "'");
					client.delete(entry.path);
				}
			}
		}
		catch (final DbxException e)
		{
			throw new CloudsyncException("Unexpected error during history cleanup", e);
		}
	}

	private RemoteItem _prepareBackupItem(final DbxEntry[] childData, final Handler handler) throws CloudsyncException
	{
		String encryptedMetadata = null;
		if (childData[1] != null)
		{
			int retryCount = 0;
			do
			{
				try
				{
					try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream())
					{
						client.getFile(childData[1].path, null, outputStream);
						encryptedMetadata = outputStream.toString("ASCII");
						break;
					}
				}
				catch (final DbxException e)
				{
					retryCount = validateException("fetch metadata", null, e, retryCount);
				}
				catch (final IOException e)
				{
					retryCount = validateException("fetch metadata", null, e, retryCount);
				}
			}
			while (true);
		}

		Long size;
		long time;

		if (childData[0].isFile())
		{
			File file = childData[0].asFile();
			size = file.numBytes;
			time = file.lastModified.getTime();
		}
		else
		{
			size = 0L;
			time = 0L;
		}

		String title = handler.getProcessedText(childData[0].name);
		String metadata = handler.getProcessedText(encryptedMetadata);

		return handler.initRemoteItem(childData[0].name, childData[0].isFolder(), title, metadata, size, FileTime.fromMillis(time));
	}

	private void _removeFromCache(final String path)
	{
		cacheFiles.remove(path);
	}

	private void _addToCache(final DbxEntry entry)
	{
		if (entry.isFolder())
		{
			cacheFiles.put(entry.path, entry);
		}
	}

	private void sleep(long duration)
	{
		try
		{
			Thread.sleep(duration);
		}
		catch (InterruptedException ex)
		{
		}
	}

	private int validateException(String name, Item item, Exception e, int count) throws CloudsyncException
	{
		if (count < RETRY_COUNT)
		{
			long currentValidate = System.currentTimeMillis();
			long current_retry_break = (currentValidate - lastValidate);
			if (lastValidate > 0 && current_retry_break < MIN_RETRY_BREAK)
			{
				sleep(MIN_RETRY_BREAK - current_retry_break);
			}

			lastValidate = currentValidate;

			count++;

			LOGGER.log(Level.WARNING, getExceptionMessage(e) + name + " - retry " + count + "/" + RETRY_COUNT);

			return count;
		}

		throw new CloudsyncException("Unexpected error during " + name + (item == null ? "" : " of " + item.getTypeName() + " '" + item.getPath() + "'"), e);
	}

	private String getExceptionMessage(Exception e)
	{
		String msg = e.getMessage();
		if (msg.contains("\n")) msg = msg.split("\n")[0];
		return "ioexception: '" + msg + "' - ";
	}

	private void initService(Handler handler)
	{
		if (isInitialized) return;

		handler.getRootItem().setRemoteIdentifier(this.backupName);
		isInitialized = true;
	}

	private String buildPath(Item item)
	{
		List<String> names = new ArrayList<>();

		do
		{
			names.add(item.getRemoteIdentifier());
		}
		while ((item = item.getParent()) != null);

		Collections.reverse(names);

		return backupRootPath + SEPARATOR + StringUtils.join(names, SEPARATOR);
	}
}
package cloudsync.connector;

import org.apache.commons.lang3.StringUtils;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.helper.CmdOptions;
import cloudsync.helper.Helper;

public class RemoteDropboxOptions
{
	private String	appKey		= null;
	private String	appSecret	= null;
	private String	tokenPath	= null;
	private String	basePath	= null;

	public RemoteDropboxOptions(CmdOptions options, String name) throws CloudsyncException
	{
		final String[] propertyNames = new String[] { "DROPBOX_APP_KEY", "DROPBOX_APP_SECRET", "DROPBOX_TOKEN_PATH", "DROPBOX_DIR" };
		for (final String propertyName : propertyNames)
		{
			if (StringUtils.isEmpty(options.getProperty(propertyName)))
			{
				throw new CloudsyncException("'" + propertyName + "' is not configured");
			}
		}

		appKey = options.getProperty("DROPBOX_APP_KEY");
		appSecret = options.getProperty("DROPBOX_APP_SECRET");
		tokenPath = Helper.preparePath(options.getProperty("DROPBOX_TOKEN_PATH"), name);
		basePath = options.getProperty("DROPBOX_DIR");
	}

	public String getAppKey()
	{
		return appKey;
	}

	public String getAppSecret()
	{
		return appSecret;
	}

	public String getTokenPath()
	{
		return tokenPath;
	}

	public String getBasePath()
	{
		return basePath;
	}
}
package cloudsync.connector;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.StringWriter;
import java.net.UnknownHostException;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.attribute.FileTime;
import java.security.GeneralSecurityException;
import java.text.DecimalFormat;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;

import cloudsync.exceptions.FileIOException;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.StringUtils;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.helper.CmdOptions;
import cloudsync.helper.Handler;
import cloudsync.helper.Helper;
import cloudsync.model.options.NetworkErrorType;
import cloudsync.model.Item;
import cloudsync.model.ItemType;
import cloudsync.model.RemoteItem;
import cloudsync.model.LocalStreamData;

import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.auth.oauth2.GoogleTokenResponse;
import com.google.api.client.googleapis.media.MediaHttpUploader;
import com.google.api.client.googleapis.media.MediaHttpUploader.UploadState;
import com.google.api.client.googleapis.media.MediaHttpUploaderProgressListener;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.HttpResponse;
import com.google.api.client.http.HttpResponseException;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.InputStreamContent;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.JsonGenerator;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.drive.Drive;
import com.google.api.services.drive.Drive.Files.Insert;
import com.google.api.services.drive.Drive.Files.Update;
import com.google.api.services.drive.DriveScopes;
import com.google.api.services.drive.model.File;
import com.google.api.services.drive.model.FileList;
import com.google.api.services.drive.model.ParentReference;
import com.google.api.services.drive.model.Property;

public class RemoteGoogleDriveConnector implements RemoteConnector
{
	private final static Logger	LOGGER						= Logger.getLogger(RemoteGoogleDriveConnector.class.getName());

	private final static String	SEPARATOR					= "/";

	private final static String	REDIRECT_URL				= "urn:ietf:wg:oauth:2.0:oob";
	private final static String	FOLDER						= "application/vnd.google-apps.folder";
	private final static String	FILE						= "application/octet-stream";

	private final static int	MIN_SEARCH_BREAK			= 5000;
	private final static int	MIN_SEARCH_RETRIES			= 12;
	private final static int	CHUNK_COUNT					= 4;															// *
	// 256kb
	private final static int	MAX_RESULTS					= 1000;
	private final static long	MIN_TOKEN_REFRESH_TIMEOUT	= 600;

	private GoogleTokenResponse	clientToken;
	private GoogleCredential	credential;
	private Drive				service;

	private Path				clientTokenPath;

	private Map<String, File>	cacheFiles;
	private Map<String, File>	cacheParents;

	private String				basePath;
	private String				backupName;
	private String				historyName;
	private Integer				historyCount;
	private long				lastValidate				= 0;
	private boolean				showProgress;
	private NetworkErrorType networkErrorBehavior;
	private int					retries;
	private int					waitretry;
	private Charset             charset;

	public RemoteGoogleDriveConnector()
	{
	}

	@Override
	public void init(String backupName, CmdOptions options) throws CloudsyncException
	{
		RemoteGoogleDriveOptions googleDriveOptions = new RemoteGoogleDriveOptions(options, backupName);
		Integer history = options.getHistory();

		showProgress = options.showProgress();
		retries = options.getRetries();
		waitretry = options.getWaitRetry() * 1000;
		networkErrorBehavior = options.getNetworkErrorBehavior();
		charset = options.getCharset();

		cacheFiles = new HashMap<>();
		cacheParents = new HashMap<>();

		this.basePath = Helper.trim(googleDriveOptions.getClientBasePath(), SEPARATOR);
		this.backupName = backupName;
		this.historyCount = history;
		this.historyName = history > 0 ? backupName + " " + new SimpleDateFormat("yyyy.MM.dd_HH.mm.ss").format(new Date()) : null;
		
		final HttpTransport httpTransport = new NetHttpTransport();
		final JsonFactory jsonFactory = new JacksonFactory();

		if (StringUtils.isNotEmpty(googleDriveOptions.getServiceAccountUser())) {
			// Service Account auth - https://developers.google.com/api-client-library/java/google-api-java-client/oauth2#service_accounts
			try {
				credential = new GoogleCredential.Builder()
					.setTransport(httpTransport)
					.setJsonFactory(jsonFactory)
					.setServiceAccountId(googleDriveOptions.getServiceAccountEmail())
					.setServiceAccountScopes(Collections.singletonList(DriveScopes.DRIVE))
					.setServiceAccountUser(googleDriveOptions.getServiceAccountUser())
					.setServiceAccountPrivateKeyFromP12File(new java.io.File(googleDriveOptions.getServiceAccountPrivateKeyP12Path()))
					.build();
			} catch (GeneralSecurityException | IOException e) {
				throw new CloudsyncException("Can't init remote google drive connector", e);
			}

		} else {
			// Installed Applications auth - https://developers.google.com/api-client-library/java/google-api-java-client/oauth2#installed_applications
			final GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(httpTransport, jsonFactory, googleDriveOptions.getClientID(),
					googleDriveOptions.getClientSecret(), Collections.singletonList(DriveScopes.DRIVE)).setAccessType("offline").setApprovalPrompt("auto").build();
			
			this.clientTokenPath = Paths.get(googleDriveOptions.getClientTokenPath());
			
			try
			{
				final String clientTokenAsJson = Files.exists(this.clientTokenPath) ? FileUtils.readFileToString(this.clientTokenPath.toFile(), charset) : null;
				
				credential = new GoogleCredential.Builder().setTransport(new NetHttpTransport()).setJsonFactory(new GsonFactory())
						.setClientSecrets(googleDriveOptions.getClientID(), googleDriveOptions.getClientSecret()).build();
				
				if (StringUtils.isEmpty(clientTokenAsJson))
				{
					final String url = flow.newAuthorizationUrl().setRedirectUri(REDIRECT_URL).build();
					System.out.println("Please open the following URL in your browser, copy the authorization code and enter below.");
					System.out.println("\n" + url + "\n");
					final String code = new BufferedReader(new InputStreamReader(System.in)).readLine().trim();
					
					clientToken = flow.newTokenRequest(code).setRedirectUri(REDIRECT_URL).execute();
					
					storeClientToken(jsonFactory);
					LOGGER.log(Level.INFO, "client token stored in '" + this.clientTokenPath + "'");
				}
				else
				{
					clientToken = jsonFactory.createJsonParser(clientTokenAsJson).parse(GoogleTokenResponse.class);
				}
				credential.setFromTokenResponse(clientToken);
			}
			catch (final IOException e)
			{
				throw new CloudsyncException("Can't init remote google drive connector", e);
			}
		}
	}

	private void storeClientToken(final JsonFactory jsonFactory) throws IOException
	{
		final StringWriter jsonTrWriter = new StringWriter();
		final JsonGenerator generator = jsonFactory.createJsonGenerator(jsonTrWriter);
		generator.serialize(clientToken);
		generator.flush();
		generator.close();

		FileUtils.writeStringToFile(clientTokenPath.toFile(), jsonTrWriter.toString(), charset);
	}

	@Override
	public void upload(final Handler handler, final Item item) throws CloudsyncException, FileIOException
	{
		initService(handler);

		String title = handler.getLocalProcessedTitle(item);
		File parentDriveItem = null;
		File driveItem;
		int retryCount = 0;
		do
		{
			try
			{
				refreshCredential();
				parentDriveItem = _getDriveItem(item.getParent());
				final ParentReference parentReference = new ParentReference();
				parentReference.setId(parentDriveItem.getId());
				driveItem = new File();
				driveItem.setTitle(title);
				driveItem.setParents(Collections.singletonList(parentReference));
				final LocalStreamData data = _prepareDriveItem(driveItem, item, handler, true);
				if (data == null)
				{
					driveItem = service.files().insert(driveItem).execute();
				}
				else
				{
					final InputStreamContent params = new InputStreamContent(FILE, data.getStream());
					params.setLength(data.getLength());
					Insert inserter = service.files().insert(driveItem, params);
					MediaHttpUploader uploader = inserter.getMediaHttpUploader();
					prepareUploader(uploader, data.getLength());
					driveItem = inserter.execute();
				}
				if (driveItem == null)
				{
					throw new CloudsyncException("Couldn't create item '" + item.getPath() + "'");
				}
				_addToCache(driveItem, null);
				item.setRemoteIdentifier(driveItem.getId());
				return;
			}
			catch (final IOException e)
			{
				if (parentDriveItem != null)
				{
					for (int i = 0; i < MIN_SEARCH_RETRIES; i++)
					{
						driveItem = _searchDriveItem(item.getParent(), title);
						if (driveItem != null)
						{
							LOGGER.log(Level.WARNING, "Google Drive IOException: " + getExceptionMessage(e) + " - found partially uploaded item - try to update");

							item.setRemoteIdentifier(driveItem.getId());
							update(handler, item, true);
							return;
						}
						LOGGER.log(Level.WARNING, "Google Drive IOException: " + getExceptionMessage(e) + " - item not uploaded - retry " + (i + 1) + "/" + MIN_SEARCH_RETRIES + " - wait "
								+ MIN_SEARCH_BREAK + " ms");
						sleep(MIN_SEARCH_BREAK);
					}
				}
				retryCount = validateException("remote upload", item, e, retryCount);
				if( retryCount == -1 ) // ignore a failing item (workaround for now)
					return;
			}
		}
		while (true);
	}

	@Override
	public void update(final Handler handler, final Item item, final boolean with_filedata) throws CloudsyncException, FileIOException
	{
		initService(handler);

		int retryCount = 0;
		do
		{
			try
			{
				refreshCredential();

				if (item.isType(ItemType.FILE))
				{
					final File _parentDriveItem = _getHistoryFolder(item);
					if (_parentDriveItem != null)
					{
						final File copyOfdriveItem = new File();
						final ParentReference _parentReference = new ParentReference();
						_parentReference.setId(_parentDriveItem.getId());
						copyOfdriveItem.setParents(Collections.singletonList(_parentReference));
						// copyOfdriveItem.setTitle(driveItem.getTitle());
						// copyOfdriveItem.setMimeType(driveItem.getMimeType());
						// copyOfdriveItem.setProperties(driveItem.getProperties());
						final File _copyOfDriveItem = service.files().copy(item.getRemoteIdentifier(), copyOfdriveItem).execute();
						if (_copyOfDriveItem == null)
						{
							throw new CloudsyncException("Couldn't make a history snapshot of item '" + item.getPath() + "'");
						}
					}
				}
				File driveItem = new File();
				final LocalStreamData data = _prepareDriveItem(driveItem, item, handler, with_filedata);
				if (data == null)
				{
					driveItem = service.files().update(item.getRemoteIdentifier(), driveItem).execute();
				}
				else
				{
					final InputStreamContent params = new InputStreamContent(FILE, data.getStream());
					params.setLength(data.getLength());
					Update updater = service.files().update(item.getRemoteIdentifier(), driveItem, params);
					MediaHttpUploader uploader = updater.getMediaHttpUploader();
					prepareUploader(uploader, data.getLength());
					driveItem = updater.execute();
				}
				if (driveItem == null)
				{
					throw new CloudsyncException("Couldn't update item '" + item.getPath() + "'");
				}
				else if (driveItem.getLabels().getTrashed())
				{
					throw new CloudsyncException("Remote item '" + item.getPath() + "' [" + driveItem.getId() + "] is trashed\ntry to run with --nocache");
				}
				_addToCache(driveItem, null);
				return;
			}
			catch (final IOException e)
			{
				retryCount = validateException("remote update", item, e, retryCount);
				if(retryCount < 0) // TODO workaround - fix this later
					retryCount = 0;
			}
		}
		while (true);
	}

	@Override
	public void remove(final Handler handler, final Item item) throws CloudsyncException
	{
		initService(handler);

		int retryCount = 0;
		do
		{
			try
			{
				refreshCredential();

				final File _parentDriveItem = _getHistoryFolder(item);
				if (_parentDriveItem != null)
				{
					final ParentReference parentReference = new ParentReference();
					parentReference.setId(_parentDriveItem.getId());
					File driveItem = new File();
					driveItem.setParents(Collections.singletonList(parentReference));
					driveItem = service.files().patch(item.getRemoteIdentifier(), driveItem).execute();
					if (driveItem == null)
					{
						throw new CloudsyncException("Couldn't make a history snapshot of item '" + item.getPath() + "'");
					}
				}
				else
				{
					service.files().delete(item.getRemoteIdentifier()).execute();
				}
				_removeFromCache(item.getRemoteIdentifier());
				return;
			}
			catch (final IOException e)
			{
				retryCount = validateException("remote remove", item, e, retryCount);
				if(retryCount < 0) // TODO workaround - fix this later
					retryCount = 0;
			}
		}
		while (true);
	}

	@Override
	public InputStream get(final Handler handler, final Item item) throws CloudsyncException
	{
		initService(handler);

		int retryCount = 0;
		do
		{
			try
			{
				refreshCredential();

				final File driveItem = _getDriveItem(item);
				final String downloadUrl = driveItem.getDownloadUrl();
				final HttpResponse resp = service.getRequestFactory().buildGetRequest(new GenericUrl(downloadUrl)).execute();
				return resp.getContent();
			}
			catch (final IOException e)
			{
				retryCount = validateException("remote get", item, e, retryCount);
				if(retryCount < 0) // TODO workaround - fix this later
					retryCount = 0;
			}
		}
		while (true);
	}

	@Override
	public List<RemoteItem> readFolder(final Handler handler, final Item parentItem) throws CloudsyncException
	{
		initService(handler);

		int retryCount = 0;
		do
		{
			try
			{
				refreshCredential();

				final List<RemoteItem> child_items = new ArrayList<>();
				final List<File> childDriveItems = _readFolder(parentItem.getRemoteIdentifier());
				for (final File child : childDriveItems)
				{
					child_items.add(_prepareBackupItem(parentItem, child, handler));
				}
				return child_items;
			}
			catch (final IOException e)
			{
				retryCount = validateException("remote fetch", parentItem, e, retryCount);
				if(retryCount < 0) // TODO workaround - fix this later
					retryCount = 0;
			}
		}
		while (true);
	}

	@Override
	public void cleanHistory(final Handler handler) throws CloudsyncException
	{
		initService(handler);

		final File backupDriveFolder = _getBackupFolder();
		final File parentDriveItem = _getDriveFolder(basePath);

		try
		{
			refreshCredential();

			final List<File> child_items = new ArrayList<>();
			for (File file : _readFolder(parentDriveItem.getId()))
			{
				if (backupDriveFolder.getId().equals(file.getId()) || !file.getTitle().startsWith(backupDriveFolder.getTitle()))
				{
					continue;
				}
				child_items.add(file);
			}

			if (child_items.size() > historyCount)
			{
				Collections.sort(child_items, new Comparator<File>()
				{
					@Override
					public int compare(final File o1, final File o2)
					{
						final long v1 = o1.getCreatedDate().getValue();
						final long v2 = o2.getCreatedDate().getValue();

						if (v1 < v2) return 1;
						if (v1 > v2) return -1;
						return 0;
					}
				});

				for (File file : child_items.subList(historyCount, child_items.size()))
				{
					LOGGER.log(Level.FINE, "cleanup history folder '" + file.getTitle() + "'");
					service.files().delete(file.getId()).execute();
				}
			}
		}
		catch (final IOException e)
		{
			throw new CloudsyncException("Unexpected error during history cleanup", e);
		}
	}

	private List<File> _readFolder(final String id) throws IOException
	{
		final List<File> child_items = new ArrayList<>();

		final String q = "'" + id + "' in parents and trashed = false";
		final Drive.Files.List request = service.files().list();
		request.setQ(q);
		request.setMaxResults(MAX_RESULTS);

		do
		{
			FileList files = request.execute();

			final List<File> result = files.getItems();
			for (final File file : result)
			{
				child_items.add(file);
			}
			request.setPageToken(files.getNextPageToken());

		}
		while (request.getPageToken() != null && request.getPageToken().length() > 0);

		return child_items;
	}

	private LocalStreamData _prepareDriveItem(final File driveItem, final Item item, final Handler handler, final boolean with_filedata) throws FileIOException
	{
		LocalStreamData data = null;
		if (with_filedata)
		{
			// "getLocalEncryptedBinary" should be called before "getMetadata"
			// to generate the needed checksum
			data = handler.getLocalProcessedBinary(item);
		}

		final String metadata = handler.getLocalProcessedMetadata(item);

		final List<Property> properties = new ArrayList<>();

		final int length = metadata.length();
		int partCounter = 0;
		// max 118 bytes (key+value)
		for (int i = 0; i < length; i += 100, partCounter++)
		{
			final String part = metadata.substring(i, Math.min(length, i + 100));
			final Property property = new Property();
			property.setKey("metadata" + partCounter);
			property.setValue(part);
			property.setVisibility("PRIVATE");
			properties.add(property);
		}

		final Property property = new Property();
		property.setKey("metadataParts");
		property.setValue(Integer.toString(partCounter));
		property.setVisibility("PRIVATE");
		properties.add(property);

		driveItem.setProperties(properties);

		driveItem.setMimeType(item.isType(ItemType.FOLDER) ? FOLDER : FILE);

		return data;
	}

	private RemoteItem _prepareBackupItem(final Item parentItem, final File driveItem, final Handler handler) throws CloudsyncException
	{
		final List<Property> properties = driveItem.getProperties();

		final Map<Integer, String> metadataMap = new HashMap<>();
		int metadataPartCount = -1;

		if (properties != null)
		{
			for (final Property property : properties)
			{
				final String key = property.getKey();
				if (!key.startsWith("metadata"))
				{
					continue;
				}

				if (key.equals("metadataParts"))
				{
					metadataPartCount = Integer.parseInt(property.getValue());
				}
				else
				{
					metadataMap.put(Integer.parseInt(key.substring(8)), property.getValue());
				}
			}
		}

		if (metadataPartCount == -1) metadataPartCount = metadataMap.size();

		final List<String> parts = new ArrayList<>();
		for (int i = 0; i < metadataPartCount; i++)
		{
			parts.add(i, metadataMap.get(i));
		}

		try
		{
			String title = handler.getProcessedText(driveItem.getTitle());
			String metadata = null;

			try
			{
				if (parts.size() > 0)
				{
					metadata = handler.getProcessedText(StringUtils.join(parts.toArray()));
				}
				else
				{
					ItemType type = driveItem.getMimeType().equals(FOLDER) ? ItemType.FOLDER : ItemType.FILE;
					LOGGER.log(Level.WARNING, "Found no metadata of " + type.getName() + " '" + parentItem.getPath() + "/" + title + "'");
				}
			}
			catch (CloudsyncException e)
			{
				ItemType type = driveItem.getMimeType().equals(FOLDER) ? ItemType.FOLDER : ItemType.FILE;
				LOGGER.log(Level.WARNING, "Can't decrypt metadata of " + type.getName() + " '" + parentItem.getPath() + "/" + title + "'");
			}

			return handler.initRemoteItem(driveItem.getId(), driveItem.getMimeType().equals(FOLDER), title, metadata, driveItem.getFileSize(),
					FileTime.fromMillis(driveItem.getCreatedDate().getValue()));
		}
		catch (CloudsyncException e)
		{
			ItemType type = driveItem.getMimeType().equals(FOLDER) ? ItemType.FOLDER : ItemType.FILE;
			throw new CloudsyncException("Can't decrypt name of " + type.getName() + " '" + driveItem.getId() + "' in '" + parentItem.getPath() + "/'");
		}
	}

	private File _searchDriveItem(final Item parentItem, String title) throws CloudsyncException
	{
		int retryCount = 0;
		do
		{
			try
			{
				final String q = "title='" + title + "' and '" + parentItem.getRemoteIdentifier() + "' in parents and trashed = false";
				final Drive.Files.List request = service.files().list();
				request.setQ(q);
				final List<File> result = request.execute().getItems();
				return result.size() == 0 ? null : result.get(0);
			}
			catch (final IOException e)
			{
				retryCount = validateException("remote search", parentItem, e, retryCount);
				if(retryCount < 0) // TODO workaround - fix this later
					retryCount = 0;
			}
		}
		while (true);
	}

	private File _getDriveItem(final Item item) throws CloudsyncException, IOException
	{
		final String id = item.getRemoteIdentifier();

		if (cacheFiles.containsKey(id))
		{
			return cacheFiles.get(id);
		}

		File driveItem;

		try
		{
			driveItem = service.files().get(id).execute();
		}
		catch (HttpResponseException e)
		{

			if (e.getStatusCode() == 404)
			{
				throw new CloudsyncException("Couldn't find remote item '" + item.getPath() + "' [" + id + "]\ntry to run with --nocache");
			}

			throw e;
		}

		if (driveItem.getLabels().getTrashed())
		{
			throw new CloudsyncException("Remote item '" + item.getPath() + "' [" + id + "] is trashed\ntry to run with --nocache");
		}

		_addToCache(driveItem, null);
		return driveItem;
	}

	private File _getHistoryFolder(final Item item) throws CloudsyncException, IOException
	{
		if (historyName == null)
		{
			return null;
		}

		final File driveRoot = _getBackupFolder();
		final List<String> parentDriveTitles = new ArrayList<>();
		Item parentItem = item;
		do
		{
			parentItem = parentItem.getParent();
			if (parentItem.getRemoteIdentifier().equals(driveRoot.getId()))
			{
				break;
			}
			final File parentDriveItem = _getDriveItem(parentItem);
			parentDriveTitles.add(0, parentDriveItem.getTitle());
		}
		while (true);

		return _getDriveFolder(basePath + SEPARATOR + historyName + SEPARATOR + StringUtils.join(parentDriveTitles, SEPARATOR));
	}

	private File _getBackupFolder() throws CloudsyncException
	{
		return _getDriveFolder(basePath + SEPARATOR + backupName);
	}

	private File _getDriveFolder(final String path) throws CloudsyncException
	{
		int retryCount = 0;
		do
		{
			try
			{
				File parentItem = service.files().get("root").execute();

				final String[] folderNames = StringUtils.split(path, SEPARATOR);

				for (final String name : folderNames)
				{
					if (cacheParents.containsKey(parentItem.getId() + ':' + name))
					{
						parentItem = cacheParents.get(parentItem.getId() + ':' + name);
					}
					else
					{
						final String q = "title='" + name + "' and '" + parentItem.getId() + "' in parents and trashed = false";

						final Drive.Files.List request = service.files().list();
						request.setQ(q);
						request.setMaxResults(MAX_RESULTS);

						do
						{
							FileList files = request.execute();

							final List<File> result = files.getItems();

							// array('q' => q))

							File _parentItem;

							if (result.size() == 0)
							{
								final File folder = new File();
								folder.setTitle(name);
								folder.setMimeType(FOLDER);
								final ParentReference parentReference = new ParentReference();
								parentReference.setId(parentItem.getId());
								folder.setParents(Collections.singletonList(parentReference));
								_parentItem = service.files().insert(folder).execute();
								if (_parentItem == null)
								{
									throw new CloudsyncException("Couldn't create folder '" + name + "'");
								}
							}
							else if (result.size() == 1)
							{
								_parentItem = result.get(0);
							}
							else
							{
								throw new CloudsyncException("base path '" + path + "' not unique");
							}

							if (!_parentItem.getMimeType().equals(FOLDER))
							{
								throw new CloudsyncException("No folder found at '" + path + "'");
							}

							_addToCache(_parentItem, parentItem);

							parentItem = _parentItem;

							request.setPageToken(files.getNextPageToken());
						}
						while (request.getPageToken() != null && request.getPageToken().length() > 0);
					}
				}
				return parentItem;
			}
			catch (final IOException e)
			{
				retryCount = validateException("remote get of '" + path + "'", null, e, retryCount);
				if(retryCount < 0) // TODO workaround - fix this later
					retryCount = 0;
			}
		}
		while (true);
	}

	private void _removeFromCache(final String id)
	{
		cacheFiles.remove(id);
	}

	private void _addToCache(final File driveItem, final File parentDriveItem)
	{
		if (driveItem.getMimeType().equals(FOLDER))
		{
			cacheFiles.put(driveItem.getId(), driveItem);
		}
		if (parentDriveItem != null)
		{
			cacheParents.put(parentDriveItem.getId() + ':' + driveItem.getTitle(), driveItem);
		}
	}

	private void sleep(long duration)
	{
		try
		{
			Thread.sleep(duration);
		}
		catch (InterruptedException ex)
		{
		}
	}

	private int validateException(String name, Item item, IOException e, int count) throws CloudsyncException
	{
		if( e instanceof GoogleJsonResponseException)
		{
			StringBuilder info = new StringBuilder("Unexpected error during ");
			info.append(name);
			if (item != null)
			{
				info.append( " of " );
				info.append( item.getInfo() );
			}
			info.append(". Remote item not found.\ntry to run with --nocache");

			switch( ((GoogleJsonResponseException)e).getStatusCode() )
			{
				case 404:
					throw new CloudsyncException( info.toString(), e);
			}
		}

		if (count < retries)
		{
			long currentValidate = System.currentTimeMillis();
			long current_retry_break = (currentValidate - lastValidate);
			if (lastValidate > 0 && current_retry_break < waitretry)
			{
				sleep(waitretry - current_retry_break);
			}

			lastValidate = currentValidate;

			count++;

			LOGGER.log(Level.WARNING, "Google Drive IOException: " + getExceptionMessage(e) + " - " + name + " - retry " + count + "/" + retries);

			return count;
		}

		if (e instanceof UnknownHostException)
		{
			LOGGER.log(Level.WARNING, "Connecting to RemoteHost " + getExceptionMessage(e) + " failed.");

			if( NetworkErrorType.ASK.equals(networkErrorBehavior) )
			{
				String answer = null;
				while (answer == null || (!"Y".equals(answer) && !"n".equals(answer)))
				{
					System.out.print("Retry again (Y/n) ");

					try
					{
						answer = new BufferedReader(new InputStreamReader(System.in)).readLine().trim();
					}
					catch (IOException _e)
					{
						break;
					}
				}

				if ("Y".equals(answer))
				{
					lastValidate = System.currentTimeMillis();
					return 0;
				}
			}
			else if( NetworkErrorType.CONTINUE.equals(networkErrorBehavior) )
			{
				lastValidate = System.currentTimeMillis();
				return -1;
			}
		}

		if (item != null)
		{
			throw new CloudsyncException("Unexpected error during " + name + " of " + item.getTypeName() + " '" + item.getPath() + "'", e);
		}
		else
		{
			throw new CloudsyncException("Unexpected error during " + name, e);
		}
	}

	private String getExceptionMessage(IOException e)
	{

		String msg = e.getMessage();
		if (msg.contains("\n")) msg = msg.split("\n")[0];
		return "'" + msg + "'";
	}

	public void initService(Handler handler) throws CloudsyncException
	{

		if (service != null) return;

		final HttpTransport httpTransport = new NetHttpTransport();
		final JsonFactory jsonFactory = new JacksonFactory();
		service = new Drive.Builder(httpTransport, jsonFactory, null)
			.setApplicationName("Backup")
			.setHttpRequestInitializer(credential)
			.build();
		if (StringUtils.isEmpty(credential.getServiceAccountId())) {
			credential.setExpiresInSeconds(MIN_TOKEN_REFRESH_TIMEOUT);
		}
		try
		{
			refreshCredential();
		}
		catch (IOException e)
		{
			throw new CloudsyncException("couldn't refresh google drive token");
		}
		handler.getRootItem().setRemoteIdentifier(_getBackupFolder().getId());
	}

	private void refreshCredential() throws IOException
	{
		if (StringUtils.isNotEmpty(credential.getServiceAccountId())) return;
		
		if (credential.getExpiresInSeconds() > MIN_TOKEN_REFRESH_TIMEOUT) return;

		if (credential.refreshToken())
		{
			clientToken.setAccessToken(credential.getAccessToken());
			clientToken.setExpiresInSeconds(credential.getExpiresInSeconds());
			clientToken.setRefreshToken(credential.getRefreshToken());

			final JsonFactory jsonFactory = new JacksonFactory();
			storeClientToken(jsonFactory);
			LOGGER.log(Level.INFO, "refreshed client token stored in '" + clientTokenPath + "'");
		}
	}

	private void prepareUploader(MediaHttpUploader uploader, long length)
	{
		int chunkSize = MediaHttpUploader.MINIMUM_CHUNK_SIZE * CHUNK_COUNT;
		int chunkCount = (int) Math.ceil(length / (double) chunkSize);

		if (showProgress && chunkCount > 1)
		{
			uploader.setDirectUploadEnabled(false);
			uploader.setChunkSize(chunkSize);
			uploader.setProgressListener(new RemoteGoogleDriveProgress(this, length));
		}
		else
		{

			uploader.setDirectUploadEnabled(true);
		}
	}

	private class RemoteGoogleDriveProgress implements MediaHttpUploaderProgressListener
	{
		private final long					length;
		private final DecimalFormat			df;
		private long						lastBytes;
		private long						lastTime;
		private final RemoteGoogleDriveConnector	connector;

		public RemoteGoogleDriveProgress(RemoteGoogleDriveConnector connector, long length)
		{
			this.length = length;
			this.connector = connector;
			df = new DecimalFormat("00");
			lastBytes = 0;
			lastTime = System.currentTimeMillis();
		}

		@Override
		public void progressChanged(MediaHttpUploader mediaHttpUploader) throws IOException
		{
			if (mediaHttpUploader == null) return;

			switch ( mediaHttpUploader.getUploadState() )
			{
				case INITIATION_COMPLETE:
					break;
				case INITIATION_STARTED:
				case MEDIA_IN_PROGRESS:

					this.connector.refreshCredential();

					double percent = mediaHttpUploader.getProgress() * 100;

					long currentTime = System.currentTimeMillis();

					String msg = "\r  " + df.format(Math.ceil(percent)) + "% (" + convertToKB(mediaHttpUploader.getNumBytesUploaded()) + " of "
							+ convertToKB(length) + " kb)";

					if (mediaHttpUploader.getUploadState().equals(UploadState.MEDIA_IN_PROGRESS))
					{
						long speed = convertToKB((mediaHttpUploader.getNumBytesUploaded() - lastBytes) / ((currentTime - lastTime) / 1000.0));
						msg += " - " + speed + " kb/s";
					}

					LOGGER.log(Level.FINEST, msg, true);

					lastTime = currentTime;
					lastBytes = mediaHttpUploader.getNumBytesUploaded();
					break;
				case MEDIA_COMPLETE:
					// System.out.println("Upload is complete!");
				default:
					break;
			}
		}

		private long convertToKB(double size)
		{
			return (long) Math.ceil(size / 1024);
		}
	}
}
package cloudsync.connector;

import org.apache.commons.lang3.StringUtils;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.helper.CmdOptions;
import cloudsync.helper.Helper;

public class RemoteGoogleDriveOptions
{
	private String	clientID						= null;
	private String	clientSecret					= null;
	private String	clientTokenPath					= null;
	private String	clientBasePath 					= null;
	private String	serviceAccountEmail				= null;
	private String	serviceAccountUser				= null;
	private String	serviceAccountPrivateKeyP12Path	= null;
	
	public RemoteGoogleDriveOptions(CmdOptions options, String name) throws CloudsyncException
	{
		clientBasePath = options.getProperty("GOOGLE_DRIVE_DIR");
		if( StringUtils.isEmpty(clientBasePath) )
		{
			throw new CloudsyncException(prepareMessage("GOOGLE_DRIVE_DIR"));
		}

		clientID = options.getProperty("GOOGLE_DRIVE_CLIENT_ID");
		clientSecret = options.getProperty("GOOGLE_DRIVE_CLIENT_SECRET");
		clientTokenPath = Helper.preparePath(options.getProperty("GOOGLE_DRIVE_CLIENT_TOKEN_PATH"), name);

		serviceAccountEmail = options.getProperty("GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL");
		serviceAccountUser = options.getProperty("GOOGLE_DRIVE_SERVICE_ACCOUNT_USER");
		serviceAccountPrivateKeyP12Path = Helper.preparePath(options.getProperty("GOOGLE_DRIVE_SERVICE_ACCOUNT_PRIVATE_KEY_P12_PATH"));

		boolean isClientTokenAccountInvalid = StringUtils.isEmpty(clientID) || StringUtils.isEmpty(clientSecret) || StringUtils.isEmpty(clientTokenPath);
		boolean isServiceAccountInvalid = StringUtils.isEmpty(serviceAccountEmail) || StringUtils.isEmpty(serviceAccountUser) ||  StringUtils.isEmpty(serviceAccountPrivateKeyP12Path);
				
		if( isClientTokenAccountInvalid && isServiceAccountInvalid )
		{
			if( !StringUtils.isEmpty(serviceAccountEmail) || !StringUtils.isEmpty(serviceAccountUser) || !StringUtils.isEmpty(serviceAccountPrivateKeyP12Path) )
			{
				if( StringUtils.isEmpty(serviceAccountEmail) ) throw new CloudsyncException(prepareMessage("GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL"));
				if( StringUtils.isEmpty(serviceAccountUser) ) throw new CloudsyncException(prepareMessage("GOOGLE_DRIVE_SERVICE_ACCOUNT_USER"));
				if( StringUtils.isEmpty(serviceAccountPrivateKeyP12Path) ) throw new CloudsyncException(prepareMessage("GOOGLE_DRIVE_SERVICE_ACCOUNT_PRIVATE_KEY_P12_PATH"));
			}
			else if( !StringUtils.isEmpty(clientID) || !StringUtils.isEmpty(clientSecret) || !StringUtils.isEmpty(clientTokenPath) )
			{
				if( StringUtils.isEmpty(clientID) ) throw new CloudsyncException(prepareMessage("GOOGLE_DRIVE_CLIENT_ID"));
				if( StringUtils.isEmpty(clientSecret) ) throw new CloudsyncException(prepareMessage("GOOGLE_DRIVE_CLIENT_SECRET"));
				if( StringUtils.isEmpty(clientTokenPath) ) throw new CloudsyncException(prepareMessage("GOOGLE_DRIVE_CLIENT_TOKEN_PATH"));
			}
			else
			{
				throw new CloudsyncException("You must configure either a 'google client token based account' or a 'google service account'");
			}
		}
	}
	
	private String prepareMessage(String name)
	{
		return "'"+name+"' is not configured";
	}

	public String getClientID()
	{
		return clientID;
	}

	public String getClientSecret()
	{
		return clientSecret;
	}

	public String getClientTokenPath()
	{
		return clientTokenPath;
	}

	public String getClientBasePath()
	{
		return clientBasePath;
	}

	public String getServiceAccountEmail() {
		return serviceAccountEmail;
	}

	public String getServiceAccountUser() {
		return serviceAccountUser;
	}

	public String getServiceAccountPrivateKeyP12Path() {
		return serviceAccountPrivateKeyP12Path;
	}
}
package cloudsync.connector;

import java.io.IOException;
import java.io.InputStream;
import java.nio.file.attribute.FileTime;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.logging.Level;
import java.util.logging.Logger;

import cloudsync.exceptions.FileIOException;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.helper.CmdOptions;
import cloudsync.helper.Handler;
import cloudsync.model.Item;
import cloudsync.model.ItemType;
import cloudsync.model.RemoteItem;
import cloudsync.model.LocalStreamData;
import java.io.File;
import java.io.FileFilter;
import java.io.FileInputStream;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardOpenOption;
import java.util.Collections;
import java.util.Comparator;
import org.apache.commons.io.FileUtils;

public class RemoteLocalFilesystemConnector implements RemoteConnector {

    private final static Logger LOGGER = Logger.getLogger(RemoteLocalFilesystemConnector.class.getName());

    /**
     * By using a format with only y/M/d, we will create an history folder per day of launch, 
     * this make more sense than a folder for each launch.
     */
    private static String HISTORY_DATE_FORMAT = "yyyy.MM.dd";
    
    final static int			MIN_SEARCH_BREAK			= 5000;
    final static int			MIN_SEARCH_RETRIES			= 12;    
    
    //private Map<String, File> cacheFiles;
    //private Map<String, File> cacheParents;

    private File remoteBackupFolder; 
    private File remoteBackupHistoryFolder;
    private String remoteTargetFolder;
    private String backupName;
    private Integer historyCount;
    private long lastValidate = 0;
    //private boolean showProgress;
    private int retries;
    private int waitretry;

    public RemoteLocalFilesystemConnector() {
    }

    @Override
    public void init(String backupName, CmdOptions options) throws CloudsyncException {
        RemoteLocalFilesystemOptions localFilesystemOptions = new RemoteLocalFilesystemOptions(options, backupName);
        Integer history = options.getHistory();

        //showProgress = options.showProgress();
        retries = options.getRetries();
        waitretry = options.getWaitRetry() * 1000;

        //cacheFiles = new HashMap<String, File>();
        //cacheParents = new HashMap<String, File>();

        this.remoteTargetFolder = localFilesystemOptions.getTargetFolder();
        this.backupName = backupName;
        this.remoteBackupFolder = new File(new File(remoteTargetFolder),this.backupName);
        this.remoteBackupFolder.mkdirs();
        this.historyCount = history;
        this.remoteBackupHistoryFolder = this.historyCount > 0 ? new File(new File(remoteTargetFolder),backupName+"_history_"+new SimpleDateFormat(HISTORY_DATE_FORMAT).format(new Date())) : null;
    }

    @Override
    public void upload(final Handler handler, final Item item) throws CloudsyncException, FileIOException {
        String title = handler.getLocalProcessedTitle(item);
        File remoteFile = new File(_getRemoteFile(item.getParent()), title);
        File remoteMetadataFile = new File(remoteFile.getParent(), remoteFile.getName() + ".metadata");
        int retryCount = 0;
        do {
            try {
                LocalStreamData data = null;

                // prepare
                if(ItemType.FOLDER.equals(item.getType())) {
                    remoteFile.mkdirs();
                }
                else {
                    data = handler.getLocalProcessedBinary(item);
                }

                final String metadata = handler.getLocalProcessedMetadata(item);

                // upload metadata
                java.nio.file.Files.write(remoteMetadataFile.toPath(), metadata.getBytes("UTF-8"),StandardOpenOption.CREATE_NEW);

                // upload file data
                if( data != null ) {
                    java.nio.file.Files.copy(data.getStream(),remoteFile.toPath());
                }
                
                // verifiy
                if (!remoteFile.exists()) {
                    throw new CloudsyncException("Couldn't create item '" + item.getPath() + "'");
                }

                if(!remoteMetadataFile.exists()) {
                    throw new CloudsyncException("Couldn't create metadata for item '" + item.getPath() + "'");
                }
                
                //_addToCache(driveItem, null);
                item.setRemoteIdentifier(remoteFile.getName());
                return;
            } catch (final IOException e) {
                for (int i = 0; i < MIN_SEARCH_RETRIES; i++) {
                    if (remoteFile.exists() || remoteMetadataFile.exists()) {
                        LOGGER.log(Level.WARNING, "RemoteLocaFilesystem IOException: " + getExceptionMessage(e) + " - found partially remote item - try to update");
                        item.setRemoteIdentifier(remoteFile.getName());
                        update(handler, item, true);
                        return;
                    }
                    LOGGER.log(Level.WARNING, "RemoteLocaFilesystem IOException: " + getExceptionMessage(e) + " - item not uploaded - retry " + (i + 1) + "/" + MIN_SEARCH_RETRIES + " - wait "
                            + MIN_SEARCH_BREAK + " ms");
                    sleep(MIN_SEARCH_BREAK);
                }
                retryCount = validateException("remote upload", item, e, retryCount);
            }
        } while (true);
    }

    @Override
    public void update(final Handler handler, final Item item, final boolean with_filedata) throws CloudsyncException, FileIOException {
        final File remoteFile = _getRemoteFile(item);
        final File remoteMetadataFile = new File(remoteFile.getParent(), remoteFile.getName() + ".metadata");
        
        int retryCount = 0;
        do {
            try {
                if (item.isType(ItemType.FILE)) {
                    if (remoteBackupHistoryFolder != null) {
                        _moveToHistory(item);
                    }
                    LocalStreamData data = handler.getLocalProcessedBinary(item);
                    java.nio.file.Files.copy(data.getStream(), remoteFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
                }
                final String metadata = handler.getLocalProcessedMetadata(item);
                java.nio.file.Files.write(remoteMetadataFile.toPath(), metadata.getBytes("UTF-8"),StandardOpenOption.TRUNCATE_EXISTING);                
                
                if (!remoteFile.exists() && ! remoteMetadataFile.exists()) {
                    throw new CloudsyncException("Couldn't update item '" + item.getPath() + "'");
                }
                //_addToCache(driveItem, null);
                return;
            } catch (final IOException e) {
                retryCount = validateException("remote update", item, e, retryCount);
            }
        } while (true);
    }
    
    private void _moveToHistory(final Item item) throws IOException,CloudsyncException {
        if (remoteBackupHistoryFolder != null) {
            final File remoteFile = _getRemoteFile(item);
            final File remoteFileMetadata = new File(remoteFile.getParent(),remoteFile.getName()+".metadata");
            final File historyRemoteFile = _getRemoteHistoryFile(item);
            final File historyRemoteFileMetadata = new File(historyRemoteFile.getParent(),historyRemoteFile.getName()+".metadata");

            remoteBackupHistoryFolder.mkdirs();
            historyRemoteFile.getParentFile().mkdirs();
            
            java.nio.file.Files.copy(remoteFile.toPath(), historyRemoteFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
            java.nio.file.Files.copy(remoteFileMetadata.toPath(), historyRemoteFileMetadata.toPath(), StandardCopyOption.REPLACE_EXISTING);
            
            if (!historyRemoteFile.exists()) {
                throw new CloudsyncException("Couldn't make a history snapshot of item '" + item.getPath() + "'");
            }
            if (!historyRemoteFileMetadata.exists()) {
                throw new CloudsyncException("Couldn't make a history snapshot of metadata item '" + item.getPath() + "'");
            }
        }
    }

    @Override
    public void remove(final Handler handler, final Item item) throws CloudsyncException {
        int retryCount = 0;
        do {
            try {
                final File remoteFile = _getRemoteFile(item);
                final File remoteFileMetadata = new File(remoteFile.getParent(),remoteFile.getParent()+".metadata");
            
                if (remoteBackupHistoryFolder != null) {
                    _moveToHistory(item);
                }
                
                remoteFile.delete();
                remoteFileMetadata.delete();
                //_removeFromCache(item.getRemoteIdentifier());
                return;
            } 
            catch (final IOException e) {
                retryCount = validateException("remote remove", item, e, retryCount);
            }
        } while (true);
    }

    @Override
    public InputStream get(final Handler handler, final Item item) throws CloudsyncException {
        int retryCount = 0;
        do {
            try {
                final File remoteFileItem = _getRemoteFile(item);
                return new FileInputStream(remoteFileItem);
            } catch (final IOException e) {
                retryCount = validateException("remote get", item, e, retryCount);
            }
        } while (true);
    }

    @Override
    public List<RemoteItem> readFolder(final Handler handler, final Item parentItem) throws CloudsyncException {
        int retryCount = 0;
        do {
            try {
                final List<RemoteItem> child_items = new ArrayList<>();
                File remoteFolder = _getRemoteFile(parentItem);
                for (final File child : remoteFolder.listFiles(new FileFilter() {
                    @Override
                    public boolean accept(File pathname) {
                        if(pathname.getName().endsWith(".metadata")) {
                            return false;
                        }
                        return true;
                    }
                })) {
                    child_items.add(_prepareBackupItem(child, handler));
                }
                return child_items;
            } 
            catch (final Exception e) {
                retryCount = validateException("remote fetch", parentItem, new IOException("Cannot read remote folder", e), retryCount);
            }
        } while (true);
    }

    @Override
    public void cleanHistory(final Handler handler) throws CloudsyncException {
        File rootTarget = remoteBackupFolder.getParentFile();
        final SimpleDateFormat sdf = new SimpleDateFormat(HISTORY_DATE_FORMAT);
        final int backupHistoryNamePrefix = (backupName+"_history_").length();
                    
        try {
            final List<File> child_items = new ArrayList<File>();
            for (File f : rootTarget.listFiles()) {
                if(f.getName().startsWith(backupName+"_history_")) {
                    sdf.parse(f.getName().substring(backupHistoryNamePrefix)); // this will throw an Unexpected error
                    child_items.add(f);
                }
            }

            if (child_items.size() > historyCount) {
                    
                    Collections.sort(child_items, new Comparator<File>() {
                        @Override
                        public int compare(final File o1, final File o2) {
                            try {
                                Date d1 = sdf.parse(o1.getName().substring(backupHistoryNamePrefix));
                                Date d2 = sdf.parse(o2.getName().substring(backupHistoryNamePrefix));
                                
                                final long v1 = d1.getTime();
                                final long v2 = d2.getTime();

                                if (v1 < v2) return 1;
                                if (v1 > v2) return -1;
                            }
                            catch(Exception e) {
                                LOGGER.severe("Error parsing datetime for history folder");
                            }
                            return 0;
                        }
                    });

                    for (File file : child_items.subList(historyCount, child_items.size())) {
                        LOGGER.log(Level.FINE, "cleanup history folder '" + file.getName() + "'");
                        FileUtils.deleteDirectory(file);
                    }
            }
        }
        catch (final Exception e) {
            throw new CloudsyncException("Unexpected error during history cleanup", e);
        }
    }
    
    private File _getRemoteFile(Item item) {
        if(item.getParent() == null) {
            return new File(remoteBackupFolder,item.getRemoteIdentifier());
        }
        return new File(_getRemoteFile(item.getParent()),item.getRemoteIdentifier());
    }
    
    private File _getRemoteHistoryFile(final Item item) throws CloudsyncException, IOException {
        if (remoteBackupHistoryFolder == null) {
            return null;
        }

        if(item.getParent() == null) {
            return new File(remoteBackupHistoryFolder,item.getRemoteIdentifier());
        }
        return new File(_getRemoteHistoryFile(item.getParent()),item.getRemoteIdentifier());
    }
    

    private RemoteItem _prepareBackupItem(final File remoteFile, final Handler handler) throws CloudsyncException {
        try {
            File remoteMetadataFile = new File(remoteFile.getParent(),remoteFile.getName() + ".metadata");
            String encryptedMetadata = remoteMetadataFile.exists() ? new String(Files.readAllBytes(remoteMetadataFile.toPath()),"UTF-8") : "";
            String metadata = encryptedMetadata.isEmpty() ? "" : handler.getProcessedText(encryptedMetadata);
            String title = handler.getProcessedText(remoteFile.getName());
            return handler.initRemoteItem(remoteFile.getName(), remoteFile.isDirectory(), title, metadata, remoteFile.length(),FileTime.fromMillis(remoteFile.lastModified()));
        } catch (Exception e) {
            throw new CloudsyncException("Can't decrypt infos about '" + remoteFile.getName());
        }
    }

    
    



    /*private void _removeFromCache(final String id) {
        cacheFiles.remove(id);
    }

    private void _addToCache(final File driveItem, final File parentDriveItem) {
        if (driveItem.isDirectory()) {
            cacheFiles.put(driveItem.getPath(), driveItem);
        }
        if (parentDriveItem != null) {
            cacheParents.put(parentDriveItem.getPath() + ':' + driveItem.getPath(), driveItem);
        }
    }*/

    private void sleep(long duration) {
        try {
            Thread.sleep(duration);
        } catch (InterruptedException ex) {
        }
    }

    private int validateException(String name, Item item, IOException e, int count) throws CloudsyncException {
        if (count < retries) {
            long currentValidate = System.currentTimeMillis();
            long current_retry_break = (currentValidate - lastValidate);
            if (lastValidate > 0 && current_retry_break < waitretry) {
                sleep(waitretry - current_retry_break);
            }

            lastValidate = currentValidate;

            count++;

            LOGGER.log(Level.WARNING, "RemoteLocaFilesystem IOException: " + getExceptionMessage(e) + " - " + name + " - retry " + count + "/" + retries);

            return count;
        }

        if (item != null) {
            throw new CloudsyncException("Unexpected error during " + name + " of " + item.getTypeName() + " '" + item.getPath() + "'", e);
        } else {
            throw new CloudsyncException("Unexpected error during " + name, e);
        }
    }

    private String getExceptionMessage(IOException e) {

        String msg = e.getMessage();
        if (msg.contains("\n")) {
            msg = msg.split("\n")[0];
        }
        return "'" + msg + "'";
    }




}
package cloudsync.connector;

import org.apache.commons.lang3.StringUtils;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.helper.CmdOptions;

public class RemoteLocalFilesystemOptions {

    private String targetFolder = null;

    public RemoteLocalFilesystemOptions(CmdOptions options, String name) throws CloudsyncException {
        targetFolder = options.getProperty("TARGET_DIR");
        if (StringUtils.isEmpty(targetFolder)) {
            throw new CloudsyncException(prepareMessage("TARGET_DIR"));
        }
    }

    private String prepareMessage(String name) {
        return "'" + name + "' is not configured";
    }

    public String getTargetFolder() {
        return targetFolder;
    }

}
package cloudsync.exceptions;

public class CloudsyncException extends Exception
{
	private static final long	serialVersionUID	= 5775239254896263691L;

	public CloudsyncException(final String message)
	{
		super(message);
	}

	public CloudsyncException(final String message, final Exception e)
	{
		super(message, e);
	}
}
package cloudsync.exceptions;

public class DataException extends Exception
{
	private static final long	serialVersionUID	= 5775239254896263691L;

	public DataException(final String message)
	{
		super(message);
	}

	public DataException(final String message, final Exception e)
	{
		super(message, e);
	}
}
package cloudsync.exceptions;

public class FileIOException extends Exception
{
	private static final long	serialVersionUID	= 5775239254896263691L;

	public FileIOException(final String message)
	{
		super(message);
	}

	public FileIOException(final String message, final Exception e)
	{
		super(message, e);
	}
}
package cloudsync.exceptions;

public class InfoException extends Exception
{
	private static final long	serialVersionUID	= 5775239254896263691L;

	public InfoException(final String message)
	{
		super(message);
	}
}
package cloudsync.exceptions;

public class UsageException extends Exception
{
	private static final long	serialVersionUID	= 5775239254896263691L;

	public UsageException(final String message)
	{
		super(message);
	}
}
package cloudsync.helper;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Properties;

import cloudsync.exceptions.InfoException;
import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.CommandLineParser;
import org.apache.commons.cli.DefaultParser;
import org.apache.commons.cli.HelpFormatter;
import org.apache.commons.cli.Option;
import org.apache.commons.cli.Options;
import org.apache.commons.cli.ParseException;
import org.apache.commons.lang3.StringUtils;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.exceptions.UsageException;
import cloudsync.model.options.NetworkErrorType;
import cloudsync.model.options.FileErrorType;
import cloudsync.model.options.ExistingType;
import cloudsync.model.Item;
import cloudsync.model.options.FollowLinkType;
import cloudsync.model.options.PermissionType;
import cloudsync.model.options.SyncType;

public class CmdOptions
{
	private final Options			options;
	private final List<Option>		positions;
	private final String[]			args;

	private String					passphrase;

	private Properties				prop;
	private SyncType				type;
	private String					path;
	private String					name;
	private Integer					history;
	private String[]				includePatterns;
	private String[]				excludePatterns;
	private String					logfilePath;
	private String					cachefilePath;
	private String					lockfilePath;
	private String					pidfilePath;
	private PermissionType permissions;
	private boolean					nocache;
	private boolean					forcestart;
	private boolean					dryrun;
	private boolean					showProgress;
	private NetworkErrorType networkErrorBehavior;
	private FileErrorType fileErrorBehavior;
	private boolean					noencryption;
	private FollowLinkType followlinks;
	private ExistingType existingBehavior;
	private String					remoteConnector;

	private int						retries;
	private int						waitretry;

	private long 					minTmpFileSize;

	public CmdOptions(final String[] args)
	{
		this.args = args;

		positions = new ArrayList<>();

		options = new Options();

		Option option = Option.builder("b")
			.hasArg()
			.argName("path")
			.desc("Create or refresh backup of <path>")
			.longOpt(SyncType.BACKUP.getName())
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder("r")
			.hasArg()
			.argName("path")
			.desc("Restore a backup into <path>")
			.longOpt(SyncType.RESTORE.getName())
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder("c")
			.hasArg()
			.argName("path")
			.desc("Repair 'cloudsync*.cache' file and put leftover file into <path>")
			.longOpt(SyncType.CLEAN.getName())
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder("l")
			.desc("List the contents of an backup")
			.longOpt(SyncType.LIST.getName())
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder("n")
			.hasArg()
			.argName("name")
			.desc("Backup name of --backup, --restore, --clean or --list")
			.longOpt("name")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.hasArg()
			.argName("path")
			.desc("Config file path. Default is './config/cloudsync.config'")
			.longOpt("config")
			.build();
		options.addOption(option);
		positions.add(option);

		String description = "How to handle symbolic links\n";
		description += "<extern> - follow symbolic links if the target is outside from the current directory hierarchy - (default)\n";
		description += "<all> - follow all symbolic links\n";
		description += "<none> - don't follow any symbolic links";
		option = Option.builder()
			.hasArg()
			.argName("extern|all|none")
			.desc(description)
			.longOpt("followlinks")
			.build();
		options.addOption(option);
		positions.add(option);

		description = "Behavior on files that exists localy during --restore\n";
		description += "<stop> - stop immediately - (default)\n";
		description += "<update> - replace file\n";
		description += "<skip> - skip file\n";
		description += "<rename> - extend the name with an autoincrement number";
		option = Option.builder()
			.hasArg()
			.argName("stop|update|skip|rename")
			.desc(description)
			.longOpt("existing")
			.build();
		options.addOption(option);
		positions.add(option);

		description = "Before remove or update a file or folder move it to a history folder.\n";
		description += "Use a maximum of <count> history folders";
		option = Option.builder()
			.hasArg()
			.argName("count")
			.desc(description)
			.longOpt("history")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.hasArg()
			.argName("pattern")
			.desc("Include content of --backup, --restore and --list if the path matches the regex based ^<pattern>$. Multiple patterns can be separated with an '|' character.")
			.longOpt("include")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.hasArg()
			.argName("pattern")
			.desc("Exclude content of --backup, --restore and --list if the path matches the regex based ^<pattern>$. Multiple patterns can be separated with an '|' character.")
			.longOpt("exclude")
			.build();
		options.addOption(option);
		positions.add(option);

		description = "Behavior how to handle acl permissions during --restore\n";
		description += "<set> - set all permissions and ownerships - (default)\n";
		description += "<ignore> - ignores all permissions and ownerships\n";
		description += "<try> - ignores invalid and not assignable permissions and ownerships\n";
		option = Option.builder()
			.hasArg()
			.argName("set|ignore|try")
			.desc(description)
			.longOpt("permissions")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.desc("Don't use 'cloudsync*.cache' file for --backup or --list (much slower)")
			.longOpt("nocache")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.desc("Ignore a existing pid file. Should only be used after a previous crashed job.")
			.longOpt("forcestart")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.desc("Don't encrypt uploaded data")
			.longOpt("noencryption")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.desc("Perform a trial run of --backup or --restore with no changes made.")
			.longOpt("dry-run")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.desc("Show progress during transfer and encryption.")
			.longOpt("progress")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.hasArg()
			.argName("number")
			.desc("Number of network operation retries before an error is thrown (default: 6).")
			.longOpt("retries")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.hasArg()
			.argName("seconds")
			.desc("Number of seconds between 2 retries (default: 10).")
			.longOpt("waitretry")
			.build();
		options.addOption(option);
		positions.add(option);

		description = "How to continue on network problems\n";
		description += "<exception> - Throw an exception - (default)\n";
		description += "<ask> - Show a command prompt (Y/n) to continue\n";
		description += "<continue> - Show a warning and continue\n";
		option = Option.builder()
			.hasArg()
			.argName("exception|ask|continue")
			.desc(description)
			.longOpt("network-error")
			.build();
		options.addOption(option);
		positions.add(option);

		description = "How to continue on blocked files or permission problems\n";
		description += "<exception> - Throw an exception - (default)\n";
		description += "<message> - Show a error log message\n";
		option = Option.builder()
			.hasArg()
			.argName("exception|message")
			.desc(description)
			.longOpt("file-error")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.hasArg()
			.argName("path")
			.desc("Log message to <path>")
			.longOpt("logfile")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.hasArg()
			.argName("path")
			.desc("Cache data to <path>")
			.longOpt("cachefile")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder()
			.hasArg()
			.argName("size")
			.desc("Minimum file size <size> in bytes to use tmp files (default: 134217728)")
			.longOpt("min_tmp_file_size")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder("v")
			.desc("Show version number")
			.longOpt("version")
			.build();
		options.addOption(option);
		positions.add(option);

		option = Option.builder("h")
			.desc("Show this help")
			.longOpt("help")
			.build();
		options.addOption(option);
		positions.add(option);
	}

	public void parse() throws UsageException, CloudsyncException, InfoException
	{
		final CommandLineParser parser = new DefaultParser();
		CommandLine cmd;
		try
		{
			cmd = parser.parse(options, args);
		}
		catch (ParseException e)
		{
			throw new UsageException(e.getMessage());
		}

		type = null;
		path = null;
		if ((path = cmd.getOptionValue(SyncType.BACKUP.getName())) != null)
		{
			type = SyncType.BACKUP;
		}
		else if ((path = cmd.getOptionValue(SyncType.RESTORE.getName())) != null)
		{
			type = SyncType.RESTORE;
		}
		else if ((path = cmd.getOptionValue(SyncType.CLEAN.getName())) != null)
		{
			type = SyncType.CLEAN;
		}
		else if (cmd.hasOption(SyncType.LIST.getName()))
		{
			type = SyncType.LIST;
		}

		String config = cmd.getOptionValue("config", "." + Item.SEPARATOR + "config" + Item.SEPARATOR + "cloudsync.config");
		if (config.startsWith("." + Item.SEPARATOR))
		{
			config = System.getProperty("user.dir") + Item.SEPARATOR + config;
		}

		boolean configValid = new File(config).isFile();
		prop = new Properties();
		try
		{
			prop.load(new FileInputStream(config));
		}
		catch (final IOException e)
		{
			configValid = false;
		}

		name = getOptionValue(cmd, "name", null);

		remoteConnector = prop.getProperty("REMOTE_CONNECTOR");

		String value = getOptionValue(cmd, "followlinks", FollowLinkType.EXTERNAL.getName());
		followlinks = FollowLinkType.fromStringIgnoreCase(value);
		value = getOptionValue(cmd, "existing", SyncType.CLEAN.equals(type) ? ExistingType.RENAME.getName() : ExistingType.STOP.getName());
		existingBehavior = ExistingType.fromStringIgnoreCase(value);
		value = getOptionValue(cmd, "permissions", PermissionType.SET.getName());
		permissions = PermissionType.fromStringIgnoreCase(value);

		history = SyncType.BACKUP.equals(type) ? Integer.parseInt(getOptionValue(cmd, "history", "0")) : 0;

		try
		{
			retries = Integer.parseInt(getOptionValue(cmd, "retries", "6"));
		}
		catch (NumberFormatException e)
		{
			retries = 0;
		}

		try
		{
			waitretry = Integer.parseInt(getOptionValue(cmd, "waitretry", "10"));
		}
		catch (NumberFormatException e)
		{
			waitretry = 0;
		}

		try
		{
			minTmpFileSize = Long.parseLong( getOptionValue(cmd, "min_tmp_file_size", "134217728" ) );
		}
		catch (NumberFormatException e)
		{
			// 128MB
			minTmpFileSize = 134217728;
		}

		value = getOptionValue(cmd, "network-error", "exception");
		networkErrorBehavior = NetworkErrorType.fromStringIgnoreCase( value);

		value = getOptionValue(cmd, "file-error", "exception");
		fileErrorBehavior = FileErrorType.fromStringIgnoreCase( value);

		nocache = cmd.hasOption("nocache") || SyncType.CLEAN.equals(type);
		forcestart = cmd.hasOption("forcestart");
		dryrun = cmd.hasOption("dry-run");
		showProgress = cmd.hasOption("progress");
		noencryption = cmd.hasOption("noencryption");

		String pattern = getOptionValue(cmd, "include", null);
		if (pattern != null) includePatterns = pattern.contains("|") ? pattern.split("\\|") : new String[] { pattern };
		pattern = getOptionValue(cmd, "exclude", null);
		if (pattern != null) excludePatterns = pattern.contains("|") ? pattern.split("\\|") : new String[] { pattern };

		if (!StringUtils.isEmpty(name))
		{
			logfilePath = Helper.preparePath(getOptionValue(cmd, "logfile", null), name);
			cachefilePath = Helper.preparePath(getOptionValue(cmd, "cachefile", null), name);
			if( !StringUtils.isEmpty(cachefilePath))
			{
				pidfilePath = cachefilePath.substring(0, cachefilePath.lastIndexOf(".")) + ".pid";
				lockfilePath = cachefilePath.substring(0, cachefilePath.lastIndexOf(".")) + ".lock";
			}
		}

		final boolean baseValid = SyncType.LIST.equals(type) || (path != null && new File(path).isDirectory());
		boolean logfileValid = logfilePath == null || new File(logfilePath).getParentFile().isDirectory();
		boolean cachefileValid = cachefilePath == null || new File(cachefilePath).getParentFile().isDirectory();

		if( cmd.hasOption("version") )
		{
			throw new InfoException("cloudsync " + getClass().getPackage().getImplementationVersion());
		}
		else if (cmd.hasOption("help") || type == null || name == null || followlinks == null || existingBehavior == null || retries == 0 || waitretry == 0
				|| permissions == null || !baseValid || !configValid || !logfileValid || !cachefileValid)
		{
			int possibleWrongOptions = cmd.getOptions().length;
			if (cmd.hasOption("help")) possibleWrongOptions--;

			List<String> messages = new ArrayList<>();
			if (possibleWrongOptions > 0)
			{
				messages.add("missing or wrong options\nerror(s):");
				if (type == null)
				{
					messages.add(" You must specifiy --backup, --restore, --list or --clean");
				}
				else if (!baseValid)
				{
					messages.add(" --" + type.getName() + " <path> not valid");
				}
				if (name == null)
				{
					messages.add(" Missing --name <name>");
				}
				if (followlinks == null)
				{
					messages.add(" Wrong --followlinks <behavior> set");
				}
				if (existingBehavior == null)
				{
					messages.add(" Wrong --existing <behavior> set");
				}
				if (retries == 0)
				{
					messages.add(" Wrong --retries <number> set");
				}
				if (waitretry == 0)
				{
					messages.add(" Wrong --waitretry <seconds> set");
				}
				if (permissions == null)
				{
					messages.add(" Wrong --permissions <behavior> set");
				}
				if (!configValid)
				{
					messages.add(" --config <path> not valid");
				}
				if (!logfileValid)
				{
					messages.add(" --logfile <path> not valid");
				}
				if (!cachefileValid)
				{
					messages.add(" --cachefile <path> not valid");
				}
			}
			else if (!cmd.hasOption("help") && !configValid)
			{
				messages.add(" No config file found");
			}
			throw new UsageException(StringUtils.join(messages, '\n'));
		}

		passphrase = prop.getProperty("PASSPHRASE");
		if (StringUtils.isEmpty(passphrase))
		{
			throw new CloudsyncException("'PASSPHRASE' is not configured");
		}
	}

	private String getOptionValue(CommandLine cmd, String key, String defaultValue)
	{
		String value = cmd.getOptionValue(key);
		if (!StringUtils.isEmpty(value)) return value;
		value = prop.getProperty(key.toUpperCase());
		if (!StringUtils.isEmpty(value)) return value;
		return defaultValue;
	}

	public void printHelp()
	{
		final HelpFormatter formatter = new HelpFormatter();
		formatter.setWidth(120);
		formatter.setOptionComparator(new Comparator<Option>()
		{

			@Override
			public int compare(Option o1, Option o2)
			{
				if (positions.indexOf(o1) < positions.indexOf(o2)) return -1;
				if (positions.indexOf(o1) > positions.indexOf(o2)) return 1;
				return 0;
			}
		});
		// formatter.setOptPrefix("");
		formatter.printHelp("cloudsync <options>", options);
	}

	public String getPath()
	{
		return path;
	}

	public SyncType getType()
	{
		return type;
	}

	public String getName()
	{
		return name;
	}

	public String[] getIncludePatterns()
	{
		return includePatterns;
	}

	public String[] getExcludePatterns()
	{
		return excludePatterns;
	}

	public Integer getHistory()
	{
		return history;
	}

	public String getLogfilePath()
	{
		return logfilePath;
	}

	public PermissionType getPermissionType()
	{
		return permissions;
	}

	public boolean getNoCache()
	{
		return nocache;
	}

	public boolean getNoEncryption()
	{
		return noencryption;
	}

	public boolean getForceStart()
	{
		return forcestart;
	}

	public boolean isDryRun()
	{
		return dryrun;
	}

	public boolean showProgress()
	{
		return showProgress;
	}

	public NetworkErrorType getNetworkErrorBehavior()
	{
		return networkErrorBehavior;
	}

	public FileErrorType getFileErrorBehavior()
	{
		return fileErrorBehavior;
	}

	public int getRetries()
	{
		return retries;
	}

	public int getWaitRetry()
	{
		return waitretry;
	}

	public long getMinTmpFileSise()
	{
		return minTmpFileSize;
	}

	public FollowLinkType getFollowLinks()
	{
		return followlinks;
	}

	public ExistingType getExistingBehavior()
	{
		return existingBehavior;
	}

	public String getProperty(String key)
	{
		return prop.getProperty(key);
	}

	public String getCacheFile()
	{
		return cachefilePath;
	}

	public String getLockFile()
	{
		return lockfilePath;
	}

	public String getPIDFile()
	{
		return pidfilePath;
	}

	public String getPassphrase()
	{
		return passphrase;
	}

	public String getRemoteConnector()
	{
		return remoteConnector;
	}

	public Charset getCharset(){ return Charset.defaultCharset(); }
}
package cloudsync.helper;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.Security;
import java.text.DecimalFormat;
import java.util.Date;
import java.util.logging.Level;
import java.util.logging.Logger;

import javax.crypto.Cipher;

import cloudsync.exceptions.FileIOException;
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.io.IOUtils;
import org.bouncycastle.bcpg.ArmoredOutputStream;
import org.bouncycastle.bcpg.CompressionAlgorithmTags;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.openpgp.PGPCompressedData;
import org.bouncycastle.openpgp.PGPCompressedDataGenerator;
import org.bouncycastle.openpgp.PGPEncryptedDataGenerator;
import org.bouncycastle.openpgp.PGPEncryptedDataList;
import org.bouncycastle.openpgp.PGPLiteralData;
import org.bouncycastle.openpgp.PGPLiteralDataGenerator;
import org.bouncycastle.openpgp.PGPObjectFactory;
import org.bouncycastle.openpgp.PGPPBEEncryptedData;
import org.bouncycastle.openpgp.PGPUtil;
import org.bouncycastle.openpgp.jcajce.JcaPGPObjectFactory;
import org.bouncycastle.openpgp.operator.jcajce.JcaPGPDigestCalculatorProviderBuilder;
import org.bouncycastle.openpgp.operator.jcajce.JcePBEDataDecryptorFactoryBuilder;
import org.bouncycastle.openpgp.operator.jcajce.JcePBEKeyEncryptionMethodGenerator;
import org.bouncycastle.openpgp.operator.jcajce.JcePGPDataEncryptorBuilder;
import org.bouncycastle.util.io.Streams;

import cloudsync.exceptions.CloudsyncException;
import cloudsync.model.Item;
import cloudsync.model.LocalStreamData;
import cloudsync.model.TempInputStream;
import org.bouncycastle.openpgp.operator.PBEDataDecryptorFactory;
import org.bouncycastle.openpgp.operator.PGPDataEncryptorBuilder;
import org.bouncycastle.openpgp.operator.bc.BcPBEDataDecryptorFactory;
import org.bouncycastle.openpgp.operator.bc.BcPBEKeyEncryptionMethodGenerator;
import org.bouncycastle.openpgp.operator.bc.BcPGPDataEncryptorBuilder;
import org.bouncycastle.openpgp.operator.bc.BcPGPDigestCalculatorProvider;

public class Crypt
{
	// private final static Logger LOGGER =
	// Logger.getLogger(Crypt.class.getName());

	private final static Logger	LOGGER				= Logger.getLogger(Crypt.class.getName());

	private final static int	BUFFER_SIZE			= 1 << 16;

	private final static int	 ENCRYPT_ALGORITHM	= PGPEncryptedDataGenerator.AES_256;
	private final static boolean ENCRYPT_ARMOR		= false;

	private final String		passphrase;
	private final boolean				showProgress;
	private final long				minTmpFileSize;
	private final boolean 			useJCE;

	public Crypt(final CmdOptions options)
	{
		passphrase = options.getPassphrase();
		showProgress = options.showProgress();
		minTmpFileSize = options.getMinTmpFileSise();

		int allowedKeyLength = 0;
		try
		{
			allowedKeyLength = Cipher.getMaxAllowedKeyLength("AES");
		}
		catch (final NoSuchAlgorithmException e)
		{
			// e.printStackTrace();
		}
		if (allowedKeyLength < Integer.MAX_VALUE)
		{
			useJCE = false;
			LOGGER.warning("Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files are not installed");
		}
		else
		{
			useJCE = true;
		}

		Security.addProvider(new BouncyCastleProvider());
	}

	public String decryptText(String text) throws CloudsyncException
	{

		text = text.replace('_', '/');
		final byte[] data = Base64.decodeBase64(text);
		try
		{
			return new String(Streams.readAll(decryptData(new ByteArrayInputStream(data))));
		}
		catch (IOException e)
		{
			throw new CloudsyncException("can't encrypt data", e);
		}
	}

	public InputStream decryptData(final InputStream stream) throws CloudsyncException
	{
		InputStream in;

		try
		{
			in = PGPUtil.getDecoderStream(stream);

			final PGPObjectFactory pgpF = new JcaPGPObjectFactory(in);
			PGPEncryptedDataList enc;
			final Object o = pgpF.nextObject();

			// the first object might be a PGP marker packet.
			if (o instanceof PGPEncryptedDataList)
			{
				enc = (PGPEncryptedDataList) o;
			}
			else
			{
				enc = (PGPEncryptedDataList) pgpF.nextObject();
			}

			final PGPPBEEncryptedData pbe = (PGPPBEEncryptedData) enc.get(0);

            PBEDataDecryptorFactory pbeDataDecryptorFactory;
            if(useJCE)
            {
	            pbeDataDecryptorFactory = new JcePBEDataDecryptorFactoryBuilder(new JcaPGPDigestCalculatorProviderBuilder().setProvider("BC").build()).setProvider("BC").build(passphrase.toCharArray());
            }
            else
            {
                BcPGPDigestCalculatorProvider pgpDigestCalculatorProvider = new BcPGPDigestCalculatorProvider();
                pbeDataDecryptorFactory = new BcPBEDataDecryptorFactory(passphrase.toCharArray(), pgpDigestCalculatorProvider);
            }
			final InputStream clear = pbe.getDataStream(pbeDataDecryptorFactory);

			PGPObjectFactory pgpFact = new JcaPGPObjectFactory(clear);

			final PGPCompressedData cData = (PGPCompressedData) pgpFact.nextObject();

			pgpFact = new JcaPGPObjectFactory(cData.getDataStream());

			final PGPLiteralData ld = (PGPLiteralData) pgpFact.nextObject();

			return ld.getInputStream();

		}
		catch (Exception e)
		{
			throw new CloudsyncException("can't decrypt data", e);
		}
	}

	public LocalStreamData encryptedBinary(final String name, final LocalStreamData data, final Item item) throws FileIOException
	{
		InputStream input = null;

		try
		{
			input = data.getStream();

			if (data.getLength() < minTmpFileSize)
			{
				final ByteArrayOutputStream output = new ByteArrayOutputStream();
				_encryptData(output, input, data.getLength(), name, item.getInfo(), ENCRYPT_ALGORITHM, ENCRYPT_ARMOR);

				final byte[] bytes = output.toByteArray();

				return new LocalStreamData(new ByteArrayInputStream(bytes), bytes.length);
			}
			else
			{
				try
				{
					File temp = File.createTempFile("encrypted", ".pgp");
					temp.deleteOnExit();
					final FileOutputStream output = new FileOutputStream(temp);
					_encryptData(output, input, data.getLength(), name, item.getInfo(), ENCRYPT_ALGORITHM, ENCRYPT_ARMOR);

					return new LocalStreamData(new TempInputStream(temp), temp.length());

				}
				catch (IOException e)
				{
					throw new FileIOException("can't encrypt data", e);
				}
			}

		}
		finally
		{
			if (input != null) IOUtils.closeQuietly(input);
		}
	}

	public String encryptText(String text) throws FileIOException
	{
		final ByteArrayOutputStream output = new ByteArrayOutputStream();
		final byte[] bytes = text.getBytes();
		_encryptData(output, new ByteArrayInputStream(bytes), bytes.length, PGPLiteralData.CONSOLE, null, ENCRYPT_ALGORITHM, ENCRYPT_ARMOR);

		text = Base64.encodeBase64String(output.toByteArray());
		text = text.replace('/', '_');
		return text;
	}

	private void _encryptData(final OutputStream output, final InputStream input, final long length, final String name, final String fileOutputInfo, final int algorithm, final boolean armor)
			throws FileIOException
	{
		OutputStream out = output;

		try
		{
			if (armor) out = new ArmoredOutputStream(out);

            PGPEncryptedDataGenerator encryptedDataGenerator;
            if(useJCE)
            {
                encryptedDataGenerator = new PGPEncryptedDataGenerator(new JcePGPDataEncryptorBuilder(algorithm).setSecureRandom(
                new SecureRandom()).setProvider("BC"));
                encryptedDataGenerator.addMethod(new JcePBEKeyEncryptionMethodGenerator(passphrase.toCharArray()).setProvider("BC"));
            }
            else
            {
                PGPDataEncryptorBuilder pgpDataEncryptorBuilder = new BcPGPDataEncryptorBuilder(algorithm);
                encryptedDataGenerator = new PGPEncryptedDataGenerator(pgpDataEncryptorBuilder);
                encryptedDataGenerator.addMethod(new BcPBEKeyEncryptionMethodGenerator(passphrase.toCharArray()));
            }
			final OutputStream encryptedData = encryptedDataGenerator.open(out, new byte[BUFFER_SIZE]);

			PGPCompressedDataGenerator compressedDataGenerator = new PGPCompressedDataGenerator(CompressionAlgorithmTags.ZIP);
			OutputStream compressedOut = compressedDataGenerator.open(encryptedData);

			final PGPLiteralDataGenerator literalDataGenerator = new PGPLiteralDataGenerator();
			final OutputStream literalOut = literalDataGenerator.open(compressedOut, PGPLiteralData.BINARY, name, new Date(), new byte[BUFFER_SIZE]);

			byte[] buffer = new byte[BUFFER_SIZE];
			int len;

			if (showProgress && output instanceof FileOutputStream && fileOutputInfo != null )
			{
				double current = 0;
				DecimalFormat df = new DecimalFormat("00");

				while ((len = input.read(buffer)) != -1)
				{
					literalOut.write(buffer, 0, len);
					current += len;
					String msg = "\r  " + df.format(Math.ceil(current * 100 / length)) + "% (" + convertToKB(current) + " of " + convertToKB(length)
							+ " kb) encrypted of " + fileOutputInfo;
					LOGGER.log(Level.FINEST, msg, true);
				}
			}
			else
			{
				while ((len = input.read(buffer)) != -1)
				{
					literalOut.write(buffer, 0, len);
				}
			}

			input.close();
			literalOut.close();
			literalDataGenerator.close();

			compressedOut.close();
			compressedDataGenerator.close();
			encryptedData.close();
			encryptedDataGenerator.close();

		}
		catch (Exception e)
		{
			throw new FileIOException("can't encrypt data", e);
		}
		finally
		{
			if (armor)
			{
				try
				{
					out.close();
				}
				catch (IOException e)
				{
					throw new FileIOException("can't close armor", e);
				}
			}

			try
			{
				output.close();
			}
			catch (IOException e)
			{
				throw new FileIOException("can't close output", e);
			}
		}
	}

	private long convertToKB(double size)
	{
		return (long) Math.ceil(size / 1024);
	}
}
package cloudsync.helper;

import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
import java.io.Reader;
import java.lang.management.ManagementFactory;
import java.lang.management.RuntimeMXBean;
import java.lang.StringBuilder;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;
import java.nio.file.attribute.FileTime;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;

import cloudsync.exceptions.FileIOException;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;
import org.apache.commons.csv.CSVRecord;
import org.apache.commons.io.FilenameUtils;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;

import cloudsync.connector.LocalFilesystemConnector;
import cloudsync.connector.RemoteConnector;
import cloudsync.exceptions.CloudsyncException;
import cloudsync.model.options.FileErrorType;
import cloudsync.model.options.ExistingType;
import cloudsync.model.Item;
import cloudsync.model.ItemType;
import cloudsync.model.options.FollowLinkType;
import cloudsync.model.options.PermissionType;
import cloudsync.model.RemoteItem;
import cloudsync.model.LocalStreamData;
import cloudsync.model.RemoteStreamData;
import cloudsync.model.options.SyncType;

public class Handler
{
	private final static Logger				LOGGER		= Logger.getLogger(Handler.class.getName());

	private final String					name;

	private final LocalFilesystemConnector	localConnection;
	private final RemoteConnector			remoteConnection;
	private final Crypt						crypt;

	private final Item						root;
	private final List<Item>				duplicates;
	private final List<Item>				invalides;
    private final List<String>			    followedLinkPaths;

private final ExistingType existingFlag;
	private final FollowLinkType followlinks;
	private final PermissionType permissionType;

	private Path							cacheFilePath;
	private Path							lockFilePath;
	private Path							pidFilePath;
	private boolean							pidCleanup	= false;

	private boolean							isLocked	= false;

	private final FileErrorType fileErrorBehavior;

	class Status
	{
		private int	create	= 0;
		private int	update	= 0;
		private int	remove	= 0;
		private int	skip	= 0;
	}

	public Handler(String name, final LocalFilesystemConnector localConnection, final RemoteConnector remoteConnection, final Crypt crypt,
			final ExistingType existingFlag, final FollowLinkType followlinks, final PermissionType permissionType, final FileErrorType fileErrorBehavior)
	{
		this.name = name;
		this.localConnection = localConnection;
		this.remoteConnection = remoteConnection;
		this.crypt = crypt;
		this.existingFlag = existingFlag;
		this.followlinks = followlinks;
		this.permissionType = permissionType;

		this.fileErrorBehavior = fileErrorBehavior;

		root = Item.getDummyRoot();
		duplicates = new ArrayList<>();
		invalides = new ArrayList<>();
		followedLinkPaths = new ArrayList<>();
	}

	public void init(SyncType synctype, String cacheFile, String lockFile, String pidFile, boolean nocache, boolean forcestart) throws CloudsyncException
	{
		cacheFilePath = Paths.get(cacheFile.replace("{name}", name));
		lockFilePath = Paths.get(lockFile.replace("{name}", name));
		pidFilePath = Paths.get(pidFile.replace("{name}", name));

		if (synctype.checkPID())
		{
			if (!forcestart && Files.exists(pidFilePath, LinkOption.NOFOLLOW_LINKS))
			{
				throw new CloudsyncException(
						"Other job is running or previous job has crashed. If you are sure that no other job is running use the option '--forcestart'");
			}

			RuntimeMXBean bean = ManagementFactory.getRuntimeMXBean();
			String jvmName = bean.getName();
			long pid = Long.valueOf(jvmName.split("@")[0]);

			try
			{
				Files.write(pidFilePath, Long.toString(pid).getBytes(), StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING);
				pidCleanup = true;
			}
			catch (IOException e)
			{
				throw new CloudsyncException("Couldn't create '" + pidFilePath.toString() + "'");
			}
		}

		if (Files.exists(lockFilePath, LinkOption.NOFOLLOW_LINKS))
		{
			LOGGER.log(Level.WARNING,
					"Found an inconsistent cache file state. Possibly previous job has crashed or duplicate files was detected. Force a cache file rebuild.");
			nocache = true;
		}

		if (!nocache && Files.exists(cacheFilePath, LinkOption.NOFOLLOW_LINKS))
		{
			LOGGER.log(Level.INFO, "load structure from cache file");
			readCSVStructure(cacheFilePath);
		}
		else
		{
			LOGGER.log(Level.INFO, "load structure from remote server");
			createLock();
			readRemoteStructure(root);
		}
		releaseLock();
	}

	@Override
	public void finalize() throws CloudsyncException
	{
		try
		{
			if (pidCleanup) Files.delete(pidFilePath);
		}
		catch (IOException e)
		{
			throw new CloudsyncException("Couldn't remove '" + pidFilePath.toString() + "'");
		}
	}

	private void createLock() throws CloudsyncException
	{
		if (isLocked) return;

		try
		{
			if (!Files.exists(lockFilePath, LinkOption.NOFOLLOW_LINKS))
			{

				Files.createFile(lockFilePath);
			}
		}
		catch (IOException e)
		{
			throw new CloudsyncException("Couldn't create '" + lockFilePath.toString() + "'");
		}

		isLocked = true;
	}

	private void releaseLock() throws CloudsyncException
	{
		if (!isLocked || duplicates.size() > 0 || invalides.size() > 0 ) return;

		try
		{
			Files.delete(lockFilePath);
		}
		catch (IOException e)
		{
			throw new CloudsyncException("Couldn't remove '" + lockFilePath.toString() + "'");
		}

		try
		{
			if (root.getChildren().size() > 0)
			{
				LOGGER.log(Level.INFO, "write structure to cache file");
				final PrintWriter out = new PrintWriter(cacheFilePath.toFile());
				final CSVPrinter csvOut = new CSVPrinter(out, CSVFormat.EXCEL);
				writeStructureToCSVPrinter(csvOut, root);
				out.close();
			}
		}
		catch (final IOException e)
		{
			throw new CloudsyncException("Can't write cache file on '" + cacheFilePath.toString() + "'", e);
		}

		isLocked = false;
	}

	private void writeStructureToCSVPrinter(final CSVPrinter out, final Item parentItem) throws IOException
	{
		for (final Item child : parentItem.getChildren().values())
		{
			out.printRecord(Arrays.asList(child.toCSVArray()));
			if (child.isType(ItemType.FOLDER))
			{
				writeStructureToCSVPrinter(out, child);
			}
		}
	}

	private void readCSVStructure(final Path cacheFilePath) throws CloudsyncException
	{
		final Map<String, Item> mapping = new HashMap<>();
		mapping.put("", root);

		try
		{
			final Reader in = new FileReader(cacheFilePath.toFile());
			final Iterable<CSVRecord> records = CSVFormat.EXCEL.parse(in);
			for (final CSVRecord record : records)
			{

				final Item item = Item.fromCSV(record);
				final String childPath = Helper.trim(record.get(0), Item.SEPARATOR);
				final String parentPath = childPath.length() == item.getName().length() ? "" : StringUtils.removeEnd(FilenameUtils.getPath(childPath),
						Item.SEPARATOR);
				mapping.put(childPath, item);
				// System.out.println(parentPath+":"+item.getName());
				Item parent = mapping.get(parentPath);
				item.setParent(parent);
				parent.addChild(item);
			}
		}
		catch (final IOException e)
		{
			throw new CloudsyncException("Can't read cache from file '" + cacheFilePath.toString() + "'", e);
		}
	}

	private void readRemoteStructure(final Item parentItem) throws CloudsyncException
	{
		Map<ItemType, Integer> status = new HashMap<>();
		readRemoteStructure(parentItem, status);
		if (status.size() > 0) LOGGER.log(Level.INFO, formatRemoteStatus(status));
	}

	private void readRemoteStructure(final Item parentItem, Map<ItemType, Integer> status) throws CloudsyncException
	{
		final List<RemoteItem> childItems = remoteConnection.readFolder(this, parentItem);

		for (final RemoteItem childItem : childItems)
		{
            if( childItem.getChecksum() == null )
            {
                LOGGER.log(Level.WARNING, "found invalides: '" + childItem.getPath());
                if (childItem.getRemoteFilesize() != null) LOGGER.log(Level.WARNING, "  size: " + childItem.getRemoteFilesize() );
                LOGGER.log(Level.WARNING, "  created: " + childItem.getRemoteCreationTime());
                invalides.add(childItem);
            }
            else
            {
                childItem.setParent(parentItem);

                final RemoteItem existingChildItem = (RemoteItem) parentItem.getChildByName(childItem.getName());
                if (existingChildItem != null)
                {
                    LOGGER.log(Level.WARNING, "found duplicate: '" + childItem.getPath());
                    String msg = "";
                    if (childItem.getRemoteFilesize() != null) msg += " " + childItem.getRemoteFilesize();
                    if (existingChildItem.getRemoteFilesize() != null) msg += " [" + existingChildItem.getRemoteFilesize() + "]";
                    if (!StringUtils.isEmpty(msg)) LOGGER.log(Level.WARNING, "  size: " + msg);
                    LOGGER.log(Level.WARNING, "  created: " + childItem.getRemoteCreationTime() + " [" + existingChildItem.getRemoteCreationTime() + "]");

                    // if childItem is newer
                    if (existingChildItem.getRemoteCreationTime().compareTo( childItem.getRemoteCreationTime() ) < 0 )
                    {
                        parentItem.addChild(childItem);
                        duplicates.add(existingChildItem);
                    }
                    else
                    {
                        duplicates.add(childItem);
                    }

                    putRemoteStatus(status, ItemType.DUPLICATE);

                }
                else
                {
                    parentItem.addChild(childItem);
                }

                if (status.size() > 0) LOGGER.log(Level.INFO, "\r  " + formatRemoteStatus(status), true);

                putRemoteStatus(status, childItem.getType());

                if (childItem.isType(ItemType.FOLDER))
                {
                    readRemoteStructure(childItem, status);
                }
            }
		}
	}

	private void putRemoteStatus(Map<ItemType, Integer> status, ItemType type)
	{
		Integer count = status.get(type);
		if (count == null) count = 0;
		count++;
		status.put(type, count);
	}

	private String formatRemoteStatus(Map<ItemType, Integer> status)
	{
		List<String> typeStatus = new ArrayList<>();
		for (ItemType type : ItemType.values())
		{
			Integer count = status.get(type);
			if (count == null) continue;
			typeStatus.add(count + " " + type.getName(count));
		}

		String lastType = typeStatus.remove(typeStatus.size() - 1);
		String message = StringUtils.join(typeStatus, ", ");
		if (message.length() > 0)
		{
			message += " and ";
		}
		message += lastType;
		return "found " + message;
	}

	private void checkErrors() throws CloudsyncException
	{
        StringBuilder message = new StringBuilder();
        
		if (duplicates.size() > 0)
		{
			message.append("found " + duplicates.size() + " duplicate item" + (duplicates.size() == 1 ? "" : "s") + ":\n\n");
			final List<Item> list = new ArrayList<>();
			for (final Item item : duplicates)
			{
				list.addAll(_flatRecursiveChildren(item));
			}
			for (final Item item : list)
			{
				message.append("  " + item.getRemoteIdentifier() + " - " + item.getPath() + "\n");
			}
		}

		if (invalides.size() > 0)
		{
            if( message.length() > 0 )
            {
                message.append("\n\n");
            }
			message.append("found " + invalides.size() + " invalid item" + (invalides.size() == 1 ? "" : "s") + ":\n\n");
			final List<Item> list = new ArrayList<>();
			for (final Item item : invalides)
			{
				list.addAll(_flatRecursiveChildren(item));
			}
			for (final Item item : list)
			{
				message.append("  " + item.getRemoteIdentifier() + " - " + item.getPath() + "\n");
			}
		}

		if( message.length() > 0 )
		{
            message.append("\n  try to run with '--clean=<path>'");

            throw new CloudsyncException(message.toString());
        }
	}

	private boolean checkPattern(String path, String[] includePatterns, String[] excludePatterns)
	{
		if (includePatterns != null)
		{
			boolean found = false;
			for (String includePattern : includePatterns)
			{
				if (path.matches("^" + includePattern + "$"))
				{
					found = true;
					break;
				}
			}
			if (!found) return false;
		}

		if (excludePatterns != null)
		{
			for (String excludePattern : excludePatterns)
			{
				if (path.matches("^" + excludePattern + "$"))
				{
					return false;
				}
			}
		}

		return true;
	}

	public void clean() throws CloudsyncException
	{
		if (duplicates.size() > 0)
		{
            _clean(duplicates);
			duplicates.clear();
		}

		if (invalides.size() > 0)
		{
            _clean(invalides);
			invalides.clear();
		}

		releaseLock();
	}
	
	private void _clean( List<Item> toClean ) throws CloudsyncException
	{
        final List<Item> list = new ArrayList<>();
        for (final Item item : toClean)
        {
            list.addAll(_flatRecursiveChildren(item));
        }
        for (final Item item : list)
        {
            localConnection.prepareUpload(this, item, ExistingType.RENAME);
            LOGGER.log(Level.FINE, "restore " + item.getTypeName() + " '" + item.getPath() + "'");
            localConnection.prepareParent(this, item);
            localConnection.upload(this, item, ExistingType.RENAME, permissionType);
        }

        Collections.reverse(list);
        for (final Item item : list)
        {
            LOGGER.log(Level.FINE, "clean " + item.getTypeName() + " '" + item.getPath() + "'");
            remoteConnection.remove(this, item);
        }
	}

	public void list(String[] includePatterns, String[] excludePatterns) throws CloudsyncException
	{
		checkErrors();

		list(includePatterns, excludePatterns, root);
	}

	private void list(String[] includePatterns, String[] excludePatterns, final Item item)
	{
		for (final Item child : item.getChildren().values())
		{
			String path = child.getPath();

			if (!checkPattern(path, includePatterns, excludePatterns)) continue;

			LOGGER.log(Level.INFO, path);

			if (child.isType(ItemType.FOLDER))
			{
				list(includePatterns, excludePatterns, child);
			}
		}
	}

	public void restore(final boolean dryRun, String[] includePatterns, String[] excludePatterns) throws CloudsyncException
	{
		checkErrors();

		restore(dryRun, includePatterns, excludePatterns, root);
	}

	private void restore(final boolean dryRun, String[] includePatterns, String[] excludePatterns, final Item item) throws CloudsyncException
	{
		for (final Item child : item.getChildren().values())
		{
			String path = child.getPath();

			if (checkPattern(path, includePatterns, excludePatterns))
			{
				localConnection.prepareUpload(this, child, existingFlag);
				LOGGER.log(Level.FINE, "restore " + child.getTypeName() + " '" + path + "'");
				if (!dryRun) localConnection.upload(this, child, existingFlag, permissionType);
			}

			if (child.isType(ItemType.FOLDER))
			{
				restore(dryRun, includePatterns, excludePatterns, child);
			}
		}
	}

	public void backup(final boolean dryRun, String[] includePatterns, String[] excludePatterns) throws CloudsyncException
	{
		checkErrors();

		final Status status = new Status();

		backup(dryRun, includePatterns, excludePatterns, root, status);

		boolean isChanged = isLocked;

		releaseLock();

		if (isChanged)
		{
			remoteConnection.cleanHistory(this);
		}

		final int total = status.create + status.update + status.skip;
		LOGGER.log(Level.INFO, "total items: " + (Integer.toString(total)));
		LOGGER.log(Level.INFO, "created items: " + (Integer.toString(status.create)));
		LOGGER.log(Level.INFO, "updated items: " + (Integer.toString(status.update)));
		LOGGER.log(Level.INFO, "removed items: " + (Integer.toString(status.remove)));
		LOGGER.log(Level.INFO, "skipped items: " + (Integer.toString(status.skip)));
	}

	private void backup(final boolean dryRun, String[] includePatterns, String[] excludePatterns, final Item remoteParentItem, final Status status)
			throws CloudsyncException
	{
		final Map<String, Item> unusedRemoteChildItems = remoteParentItem.getChildren();

		for (File localChildFile : localConnection.readFolder(remoteParentItem))
		{
			String filePath = localChildFile.getAbsolutePath();
			if (!checkPattern(filePath, includePatterns, excludePatterns)) continue;

			String backupPath = filePath;
			Item remoteChildItem = null;
			try
			{
				Item localChildItem = localConnection.getItem( localChildFile, followlinks, followedLinkPaths );
				localChildItem.setParent(remoteParentItem);

				backupPath = localChildItem.getPath();

				remoteChildItem = remoteParentItem.getChildByName(localChildItem.getName());

				if (remoteChildItem == null)
				{
					remoteChildItem = localChildItem;
					LOGGER.log(Level.FINE, "create " + remoteChildItem.getTypeName() + " '" + backupPath + "'");
					if (!dryRun)
					{
						createLock();
						remoteConnection.upload(this, remoteChildItem);
					}
					remoteParentItem.addChild(remoteChildItem);
					status.create++;
				}
				else
				{
					if (remoteChildItem.isTypeChanged(localChildItem))
					{
						LOGGER.log(Level.FINE, "remove " + remoteChildItem.getTypeName() + " '" + backupPath + "'");
						if (!dryRun)
						{
							createLock();
							remoteConnection.remove(this, remoteChildItem);
						}
						status.remove++;

						remoteChildItem = localChildItem;
						LOGGER.log(Level.FINE, "create " + remoteChildItem.getTypeName() + " '" + backupPath + "'");
						if (!dryRun)
						{
							createLock();
							remoteConnection.upload(this, remoteChildItem);
						}
						remoteParentItem.addChild(remoteChildItem);
						status.create++;
					}
					// check filesize and modify time
					else if (remoteChildItem.isMetadataChanged(localChildItem))
					{
						final boolean isFiledataChanged = localChildItem.isFiledataChanged(remoteChildItem);
						remoteChildItem.update(localChildItem);
						List<String> types = new ArrayList<>();
						if (isFiledataChanged) types.add("data,attributes");
						else if (!isFiledataChanged) types.add("attributes");
						if (remoteChildItem.isMetadataFormatChanged()) types.add("format");
						LOGGER.log(Level.FINE, "update " + remoteChildItem.getTypeName() + " '" + backupPath + "' [" + StringUtils.join(types, ",") + "]");
						if (!dryRun)
						{
							createLock();
							remoteConnection.update(this, remoteChildItem, isFiledataChanged);
						}
						status.update++;
					}
					else
					{
						status.skip++;
					}
				}

				try
				{
					// refresh Metadata
					Item _localChildItem = localConnection.getItem(localChildFile, followlinks, followedLinkPaths );
					if (_localChildItem.isMetadataChanged(localChildItem))
					{
						LOGGER.log(Level.WARNING, localChildItem.getTypeName() + " '" + backupPath + "' was changed during update.");
					}
				}
				catch (FileIOException e)
				{
					LOGGER.log(Level.WARNING, localChildItem.getTypeName() + " '" + backupPath + "' was removed during update.");
				}

				unusedRemoteChildItems.remove(remoteChildItem.getName());

				if (remoteChildItem.isType(ItemType.FOLDER))
				{
					backup(dryRun, includePatterns, excludePatterns, remoteChildItem, status);
				}
			}
			catch (FileIOException e)
			{
				status.skip++;
				if(FileErrorType.MESSAGE.equals( fileErrorBehavior))
				{
					LOGGER.log(Level.SEVERE, "Skip '" + backupPath + "'. " + e.getMessage());
					if( remoteChildItem != null ) {
						unusedRemoteChildItems.remove(remoteChildItem.getName());
					}
				}
				else
				{
					throw new CloudsyncException("Skip '" + backupPath + "'", e);
				}
			}
		}

		for (final Item item : unusedRemoteChildItems.values())
		{
			LOGGER.log(Level.FINE, "remove " + item.getTypeName() + " '" + item.getPath() + "'");
			remoteParentItem.removeChild(item);
			if (!dryRun)
			{
				createLock();
				remoteConnection.remove(this, item);
			}
			status.remove++;
		}
	}

	private List<Item> _flatRecursiveChildren(final Item parentItem)
	{

		final List<Item> list = new ArrayList<>();
		list.add(parentItem);

		if (parentItem.isType(ItemType.FOLDER))
		{
			for (final Item childItem : parentItem.getChildren().values())
			{
				list.addAll(_flatRecursiveChildren(childItem));
			}
		}

		return list;
	}

	public Item getRootItem()
	{
		return root;
	}

	public LocalStreamData getLocalProcessedBinary(final Item item) throws FileIOException
	{
		LocalStreamData data = localConnection.getFileBinary(item);

		if (data != null && crypt != null ) data = crypt.encryptedBinary(item.getName(), data, item);

		return data;
	}

	public String getLocalProcessedMetadata(final Item item) throws FileIOException
	{
		String metadata = item.getMetadata(this);
		return crypt != null ? crypt.encryptText(metadata) : metadata;
	}

	public String getLocalProcessedTitle(final Item item) throws FileIOException
	{
		return crypt != null ? crypt.encryptText(item.getName()) : item.getName();
	}

	public RemoteItem initRemoteItem(String remoteIdentifier, boolean isFolder, String title, String metadata, Long remoteFilesize, FileTime remoteCreationtime)
	{
		return Item.fromMetadata(remoteIdentifier, isFolder, title, metadata, remoteFilesize, remoteCreationtime);
	}

	public RemoteStreamData getRemoteProcessedBinary(Item item) throws CloudsyncException
	{
		InputStream stream = remoteConnection.get(this, item);
		
		if( crypt != null )
		{
			try
			{
				return new RemoteStreamData(stream,crypt.decryptData(stream));
			}
			catch(Exception e)
			{
				if( stream != null ) IOUtils.closeQuietly(stream);
				throw e;
			}
		}
		else
		{
			return new RemoteStreamData(null,stream);
		}
	}

	public String getProcessedText(final String text) throws CloudsyncException
	{
		return crypt != null ? crypt.decryptText(text) : text;
	}
}
package cloudsync.helper;

import org.apache.commons.lang3.StringUtils;

import cloudsync.model.Item;

public class Helper
{
	public static String trim(String text, final String character)
	{
		text = StringUtils.removeStart(text, character);
		text = StringUtils.removeEnd(text, character);
		return text;
	}

	public static String preparePath(String path)
	{
		return preparePath(path, "");
	}
	
	public static String preparePath(String path, String name)
	{
		if( path == null ) return null;
		
		if (path.startsWith("." + Item.SEPARATOR))
		{
			path = System.getProperty("user.dir") + path.substring(1);
		}
		return path.replace("{name}", name);
	}
}
package cloudsync.logging;

import java.text.SimpleDateFormat;
import java.util.logging.ConsoleHandler;
import java.util.logging.LogRecord;

import org.apache.commons.lang3.StringUtils;

public class LogconsoleHandler extends ConsoleHandler
{
	private boolean					keepCurrentLine;
	private int						sizeCurrentLine;
	final static private SimpleDateFormat	sdf	= new SimpleDateFormat("yyyy.MM.dd HH:mm:ss");

	@Override
	public void publish(LogRecord record)
	{
		if (record.getParameters() == null)
		{
			if (keepCurrentLine)
			{
				keepCurrentLine = false;
				sizeCurrentLine = 0;
				System.out.print("\r");
			}
			System.out.println(LogfileFormatter.formatRecord(record, sdf));

		}
		else
		{
			int _sizeCurrentLine = record.getMessage().length();

			if (_sizeCurrentLine < sizeCurrentLine) System.out.print("\r" + StringUtils.repeat(' ', sizeCurrentLine));

			sizeCurrentLine = _sizeCurrentLine;
			keepCurrentLine = true;
			System.out.print(record.getMessage());
		}
	}
}
package cloudsync.logging;

import java.text.SimpleDateFormat;
import java.util.logging.Formatter;
import java.util.logging.LogRecord;

public class LogfileFormatter extends Formatter
{
	final static private SimpleDateFormat	sdf	= new SimpleDateFormat("yyyy.MM.dd HH:mm:ss");

	@Override
	public String format(LogRecord record)
	{
		return formatRecord(record, sdf) + "\n";
	}

	public static String formatRecord(LogRecord record, SimpleDateFormat sdf)
	{
		return "[" + sdf.format(record.getMillis()) + "] - " + record.getLevel().getName() + " - " + record.getMessage();
	}
}
package cloudsync.logging;

import java.io.IOException;
import java.util.logging.FileHandler;
import java.util.logging.LogRecord;

public class LogfileHandler extends FileHandler
{
	public LogfileHandler(String pattern) throws IOException, SecurityException
	{
		super(pattern);
	}

	@Override
	public void publish(LogRecord record)
	{
		if (record.getParameters() != null) return;

		super.publish(record);
	}
}
package cloudsync.model;

import java.io.File;
import java.nio.file.attribute.FileTime;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;

import cloudsync.exceptions.FileIOException;
import org.apache.commons.csv.CSVRecord;
import org.apache.commons.io.FilenameUtils;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;

import cloudsync.helper.Handler;

public class Item
{
	public final static Integer	METADATA_VERSION	= 1;

	public final static String	SEPARATOR			= File.separator;

	public final static String	ATTRIBUTE_POSIX		= "posix";
	public final static String	ATTRIBUTE_DOS		= "dos";
	public final static String	ATTRIBUTE_OWNER		= "owner";
	public final static String	ATTRIBUTE_ACL		= "acl";

	private final static String	METADATA_SEPARATOR	= ":";
	private final static String	ATTRIBUTE_SEPARATOR	= "|";

	private Item				parent;

	protected String			name;
	protected String			remoteIdentifier;
	protected ItemType			type;
	private Long				filesize;
	private Long				creationtime;
	private Long				modifytime;
	private Long				accesstime;
	private String[]			attributes;

	private String				checksum;

	private boolean				needsMetadataUpgrade;

	protected Map<String, Item>	children;

	protected Item()
	{
	}

	public static Item getDummyRoot()
	{
		Item item = new Item();
		item.name = "";
		item.remoteIdentifier = "";
		item.type = ItemType.FOLDER;
		item.children = new HashMap<>();
		return item;
	}

	public static Item fromCSV(final CSVRecord values)
	{
		final String name = FilenameUtils.getName(values.get(0));
		final String remoteIndentifier = values.get(1);

		String[] metadata = new String[values.size() - 2];
		for (int i = 2; i < values.size(); i++)
		{
			metadata[i - 2] = values.get(i);
		}

		return initItem(new Item(), name, remoteIndentifier, metadata);
	}

	public String[] toCSVArray()
	{
		return ArrayUtils.addAll(new String[] { getPath(), remoteIdentifier }, getDataArray());
	}

	public static RemoteItem fromMetadata(final String remoteIdentifier, final boolean isFolder, final String name, final String metadata, Long remoteFilesize,
			FileTime remoteCreationtime)
	{
		RemoteItem item = new RemoteItem(remoteFilesize, remoteCreationtime);

		if (!StringUtils.isEmpty(metadata))
		{
			item = (RemoteItem) initItem(item, name, remoteIdentifier, metadata.split(METADATA_SEPARATOR));
		}
		else
		{
			item.name = name;
			item.remoteIdentifier = remoteIdentifier;
			item.type = isFolder ? ItemType.FOLDER : ItemType.FILE;
			if (item.type.equals(ItemType.FOLDER))
			{
				item.children = new HashMap<>();
			}
		}

		return item;
	}

	public String getMetadata(Handler handler) throws FileIOException
	{
		if (needsMetadataUpgrade)
		{
			if (checksum == null)
			{
				// force a checksum update
				handler.getLocalProcessedBinary(this);
			}
		}

		return StringUtils.join(getDataArray(), METADATA_SEPARATOR);
	}

	public static Item fromLocalData(String name, ItemType type, Long filesize, FileTime creationtime, FileTime modifytime, FileTime accesstime,
			Map<String, String[]> map)
	{
		Item item = new Item();
		item.name = name;
		item.type = type;
		item.filesize = filesize;
		item.creationtime = creationtime.to(TimeUnit.SECONDS);
		item.modifytime = modifytime.to(TimeUnit.SECONDS);
		item.accesstime = accesstime.to(TimeUnit.SECONDS);
		item.attributes = convertToAttributes(map);
		if (item.type.equals(ItemType.FOLDER))
		{
			item.children = new HashMap<>();
		}
		return item;
	}

	private static Item initItem(final Item item, final String name, final String remoteIdentifier, final String[] metadata)
	{
		int metadataVersion = 0;
		if (metadata.length != 9 || metadata[8].contains(ATTRIBUTE_SEPARATOR))
		{
			metadataVersion = Integer.parseInt(metadata[0]);
		}

		ItemType type;
		Long filesize;
		Long creationtime;
		Long modifytime;
		Long accesstime;
		String checksum;
		String[] attributes;

		switch ( metadataVersion )
		{
			case 0:
				type = ItemType.fromString(metadata[0]);
				filesize = StringUtils.isEmpty(metadata[1]) ? null : Long.parseLong(metadata[1]);
				creationtime = StringUtils.isEmpty(metadata[2]) ? null : Long.parseLong(metadata[2]);
				modifytime = StringUtils.isEmpty(metadata[3]) ? null : Long.parseLong(metadata[3]);
				accesstime = StringUtils.isEmpty(metadata[4]) ? null : Long.parseLong(metadata[4]);

				final String group = StringUtils.isEmpty(metadata[5]) ? null : metadata[5];
				final String user = StringUtils.isEmpty(metadata[6]) ? null : metadata[6];
				final Integer permissions = StringUtils.isEmpty(metadata[7]) ? null : Integer.parseInt(metadata[7]);
				Map<String, String[]> map = new HashMap<>();
				if (permissions == null) map.put(ATTRIBUTE_POSIX, new String[] { group, user });
				else map.put(ATTRIBUTE_POSIX, new String[] { group, user, permissions.toString() });
				attributes = convertToAttributes(map);

				checksum = metadata[8];
				break;
			default:

				type = ItemType.fromString(metadata[1]);
				filesize = StringUtils.isEmpty(metadata[2]) ? null : Long.parseLong(metadata[2]);
				creationtime = StringUtils.isEmpty(metadata[3]) ? null : Long.parseLong(metadata[3]);
				modifytime = StringUtils.isEmpty(metadata[4]) ? null : Long.parseLong(metadata[4]);
				accesstime = StringUtils.isEmpty(metadata[5]) ? null : Long.parseLong(metadata[5]);
				checksum = metadata[6];
				attributes = ArrayUtils.subarray(metadata, 7, metadata.length);

				break;
		}

		item.name = name;
		item.remoteIdentifier = remoteIdentifier;
		item.type = type;
		item.filesize = filesize;
		item.creationtime = creationtime;
		item.modifytime = modifytime;
		item.accesstime = accesstime;
		item.attributes = attributes;
		if (ItemType.FOLDER.equals(item.type))
		{
			item.children = new HashMap<>();
		}
		item.checksum = checksum;
		item.needsMetadataUpgrade = metadataVersion != METADATA_VERSION;
		return item;
	}

	private String[] getDataArray()
	{
		return ArrayUtils.addAll(
				new String[] { METADATA_VERSION.toString(), type.toString(), filesize != null ? filesize.toString() : null,
						creationtime != null ? creationtime.toString() : null, modifytime != null ? modifytime.toString() : null,
						accesstime != null ? accesstime.toString() : null, checksum }, attributes);
	}

	private static Map<String, String[]> convertToMap(String[] attributes)
	{
		Map<String, String[]> map = new HashMap<>();
		for (String row : attributes)
		{
			String[] values = StringUtils.splitPreserveAllTokens(row, ATTRIBUTE_SEPARATOR);
			map.put(values[0], ArrayUtils.subarray(values, 1, values.length));
		}
		return map;
	}

	private static String[] convertToAttributes(Map<String, String[]> map)
	{
		List<String> list = new ArrayList<>();
		for (String type : map.keySet())
		{
			list.add(type + ATTRIBUTE_SEPARATOR + StringUtils.join(map.get(type), ATTRIBUTE_SEPARATOR));
		}
		String[] arr = new String[list.size()];
		arr = list.toArray(arr);
		return arr;
	}

	public void setParent(final Item parent)
	{
		this.parent = parent;
	}

	public Item getParent()
	{
		return parent;
	}

	public void setRemoteIdentifier(final String remoteIdentifier)
	{
		this.remoteIdentifier = remoteIdentifier;
	}

	public void addChild(final Item child)
	{
		children.put(child.name, child);
	}

	public void removeChild(final Item child)
	{
		children.remove(child.name);
	}

	public Item getChildByName(final String name)
	{
		return children.get(name);
	}

	public Map<String, Item> getChildren()
	{
		return new HashMap<>(children);
	}

	public String getTypeName()
	{
		return type.getName();
	}

	public ItemType getType()
	{
		return type;
	}

	public boolean isTypeChanged(final Item item)
	{
		return !type.equals(item.type);
	}

	public boolean isFiledataChanged(final Item item)
	{
		if (type.equals(ItemType.FILE) || type.equals(ItemType.LINK))
		{
			if (isChanged(filesize, item.filesize))
			{
				return true;
			}
			if (isChanged(creationtime, item.creationtime))
			{
				return true;
			}
			if (isChanged(modifytime, item.modifytime))
			{
				return true;
			}
			/*
			 * if (isChanged(accesstime, item.accesstime)) { return true; }
			 */
		}
		return false;
	}

	public boolean isMetadataChanged(final Item item)
	{
		if (isChanged(filesize, item.filesize))
		{
			return true;
		}
		if (isChanged(creationtime, item.creationtime))
		{
			return true;
		}
		if (isChanged(modifytime, item.modifytime))
		{
			return true;
		}
		/*
		 * if (isChanged(accesstime, item.accesstime)) { return true; }
		 */
		if ((attributes == null) != (item.attributes == null) || (attributes != null && !Arrays.equals(attributes, item.attributes)))
		{
			return true;
		}
		if (needsMetadataUpgrade)
		{
			return true;
		}
		return false;
	}

	public boolean isMetadataFormatChanged()
	{
		return needsMetadataUpgrade;
	}

	private boolean isChanged(final Object o1, final Object o2)
	{
		return (o1 == null) != (o2 == null) || (o1 != null && !o1.equals(o2));
	}

	public void update(final Item item)
	{
		type = item.type;
		filesize = item.filesize;
		modifytime = item.modifytime;
		creationtime = item.creationtime;
		attributes = item.attributes;
	}

	public String getPath()
	{
		String path = "";
		if (parent != null)
		{
			path += parent.getPath();
		}
		if (!path.isEmpty())
		{
			path += Item.SEPARATOR;
		}
		path += name;
		return path;
	}

	public String getName()
	{
		return name;
	}

	public void setName(final String name)
	{
		this.name = name;
	}

	public void setChecksum(final String checksum)
	{
		this.checksum = checksum;
	}

	public String getChecksum()
	{
		return this.checksum;
	}

	public String getRemoteIdentifier()
	{
		return remoteIdentifier;
	}

	public boolean isType(final ItemType type)
	{
		return this.type.equals(type);
	}

	public Map<String, String[]> getAttributes()
	{
		return convertToMap(attributes);
	}

	public Long getFilesize()
	{
		return filesize;
	}

	public FileTime getCreationTime()
	{

		return FileTime.from(creationtime, TimeUnit.SECONDS);
	}

	public FileTime getModifyTime()
	{

		return FileTime.from(modifytime, TimeUnit.SECONDS);
	}

	public FileTime getAccessTime()
	{

		return FileTime.from(accesstime, TimeUnit.SECONDS);
	}

	public String getInfo()
	{
		StringBuilder info = new StringBuilder();
		info.append( getTypeName() );
		info.append( " '" );
		info.append( getPath() );
		info.append( "'" );
		return info.toString();
	}
}
package cloudsync.model;

public enum ItemType
{
	UNKNOWN(0, "unknown", "unknown"), FOLDER(1, "folder", "folder"), FILE(2, "file", "files"), LINK(3, "link", "links"), DUPLICATE(99, "duplicate",
			"duplicates");

	private final Integer	value;
	private final String	name;
	private final String	namePlural;

	ItemType(final Integer value, final String name, final String namePlural)
	{
		this.value = value;
		this.name = name;
		this.namePlural = namePlural;
	}

	@Override
	public String toString()
	{
		return value.toString();
	}

	public String getName()
	{
		return name;
	}

	public String getName(Integer count)
	{
		return count == 1 ? name : namePlural;
	}

	public static ItemType fromString(final String value)
	{
		final Integer intValue = Integer.parseInt(value);

		for (final ItemType type : ItemType.values())
		{
			if (!type.value.equals(intValue))
			{
				continue;
			}

			return type;
		}

		return null;
	}
}
package cloudsync.model;

import java.io.InputStream;

public class LocalStreamData
{
	private final InputStream	data;
	private final long			length;

	public LocalStreamData(InputStream data, long length)
	{
		this.data = data;
		this.length = length;
	}

	public InputStream getStream()
	{
		return data;
	}

	public long getLength()
	{
		return length;
	}
}
package cloudsync.model;

import java.nio.file.attribute.FileTime;

public class RemoteItem extends Item
{
	private final Long		remoteFilesize;
	private final FileTime	remoteCreationtime;

	public RemoteItem(Long remoteFilesize, FileTime remoteCreationtime)
	{
		super();

		this.remoteFilesize = remoteFilesize;
		this.remoteCreationtime = remoteCreationtime;
	}

	public Long getRemoteFilesize()
	{
		return remoteFilesize;
	}

	public FileTime getRemoteCreationTime()
	{
		return remoteCreationtime;
	}
}
package cloudsync.model;

import java.io.InputStream;

import org.apache.commons.io.IOUtils;

public class RemoteStreamData
{
	private final InputStream	encryptedStream;
	private final InputStream	decryptedStream;

	public RemoteStreamData(InputStream encryptedStream, InputStream decryptedStream)
	{
		this.encryptedStream = encryptedStream;
		this.decryptedStream = decryptedStream;
	}

	public InputStream getDecryptedStream()
	{
		return decryptedStream;
	}
	
	public void close()
	{
		if (encryptedStream != null) IOUtils.closeQuietly(encryptedStream);
		if (decryptedStream != null) IOUtils.closeQuietly(decryptedStream);
	}
}
package cloudsync.model;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;

public class TempInputStream extends FileInputStream
{
	private File	file;

	public TempInputStream(File file) throws FileNotFoundException
	{
		super(file);
		this.file = file;
	}

	@Override
	public void close() throws IOException
	{
		super.close();
		this.file.delete();
	}

}
package cloudsync.model.options;

public enum ExistingType
{
	STOP("stop"), UPDATE("update"), SKIP("skip"), RENAME("rename");

	private final String name;

	ExistingType(final String name)
	{
		this.name = name;
	}

	public String getName()
	{
		return name;
	}

	public static ExistingType fromStringIgnoreCase(final String name)
	{
		if(name == null)
		{
			return null;
		}

		for (final ExistingType type : ExistingType.values())
		{
			if(!type.name.equals(name.toLowerCase()))
			{
				continue;
			}

			return type;
		}

		return null;
	}
}
package cloudsync.model.options;

public enum FileErrorType
{
	EXCEPTION("exception"), MESSAGE("message");

	private final String name;

	FileErrorType(final String name)
	{
		this.name = name;
	}

	public String getName()
	{
		return name;
	}

	public static FileErrorType fromStringIgnoreCase(final String name)
	{
		if(name == null)
		{
			return null;
		}

		for (final FileErrorType type : FileErrorType.values())
		{
			if(!type.name.equals(name.toLowerCase()))
			{
				continue;
			}

			return type;
		}

		return null;
	}
}
package cloudsync.model.options;

public enum FollowLinkType
{
	EXTERNAL("external"), NONE("none"), ALL("all");

	private final String	name;

	FollowLinkType(final String name)
	{
		this.name = name;
	}

	public String getName()
	{
		return name;
	}

	public static FollowLinkType fromStringIgnoreCase(final String name)
	{
		if(name == null)
		{
			return null;
		}

		for (final FollowLinkType type : FollowLinkType.values())
		{
			if (!type.name.equals(name.toLowerCase()))
			{
				continue;
			}

			return type;
		}

		return null;
	}
}
package cloudsync.model.options;

public enum NetworkErrorType
{
	EXCEPTION("exception"), ASK("ask"), CONTINUE("continue");

	private final String name;

	NetworkErrorType(final String name)
	{
		this.name = name;
	}

	public String getName()
	{
		return name;
	}

	public static NetworkErrorType fromStringIgnoreCase(final String name)
	{
		if(name == null)
		{
			return null;
		}

		for (final NetworkErrorType type : NetworkErrorType.values())
		{
			if(!type.name.equals(name.toLowerCase()))
			{
				continue;
			}

			return type;
		}

		return null;
	}
}
package cloudsync.model.options;

public enum PermissionType
{
	SET("set"), IGNORE("ignore"), TRY("try");

	private final String	name;

	PermissionType(final String name)
	{
		this.name = name;
	}

	public String getName()
	{
		return name;
	}

	public static PermissionType fromStringIgnoreCase(final String name)
	{
		if(name == null)
		{
			return null;
		}

		for (final PermissionType type : PermissionType.values())
		{
			if (!type.name.equals(name.toLowerCase()))
			{
				continue;
			}

			return type;
		}

		return null;
	}
}
package cloudsync.model.options;

public enum SyncType
{
	BACKUP("backup", true), RESTORE("restore", false), LIST("list", false), CLEAN("clean", true);

	private final String	name;
	private final boolean	checkpid;

	SyncType(final String name, final boolean checkpid)
	{
		this.name = name;
		this.checkpid = checkpid;
	}

	public String getName()
	{
		return name;
	}

	public boolean checkPID()
	{
		return checkpid;
	}
}
package cloudsync;

public class FilesystemHelper {
    private static String OS = System.getProperty("os.name").toLowerCase();

    public static boolean isWindows() {
        return (OS.indexOf("win") >= 0);
    }

    public static String fixPathSeparators(String path) {
        if (isWindows()) {
            return path.replace("\\", "\\\\");
        }
        return path;
    }
}
package cloudsync;


import cloudsync.exceptions.CloudsyncException;
import cloudsync.exceptions.FileIOException;
import cloudsync.exceptions.InfoException;
import cloudsync.exceptions.UsageException;
import cloudsync.helper.CmdOptions;
import cloudsync.helper.Crypt;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.StandardOpenOption;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.Random;
import org.apache.commons.cli.ParseException;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import static org.junit.Assert.*;


public class RemoteLocalFilesystemTest {
    
    public RemoteLocalFilesystemTest() {
    }
    
    @BeforeClass
    public static void setUpClass() {
    }
    
    @AfterClass
    public static void tearDownClass() {
    }
    
    @Before
    public void setUp() throws IOException {
    }
    
    @After
    public void tearDown() throws IOException {
    }

    
    
    /**
     * Generate a random hierarchy, backup then restore it. 
     * Check for equality between the origin hierarchy and restored one.
     */
    @Test
    public void test1() throws IOException, ParseException {
        File rootFolder = createRandomHierarchie();
        
        File targetLocalRemoteFolder = Files.createTempDirectory("targetRemoteFolder").toFile();
        File restoreFolder = new File(rootFolder.getParent(),rootFolder.getName()+"_restore");
        File configFile = Files.createTempFile("test1Config", ".config").toFile();

        String targetLocalRemoteFolderPath = FilesystemHelper.fixPathSeparators(targetLocalRemoteFolder.getAbsolutePath());

        String config = "REMOTE_CONNECTOR=LocalFilesystem";
        config += "\n"+ "PASSPHRASE=1234567";
        config += "\n"+ "TARGET_DIR=" + targetLocalRemoteFolderPath;
        config += "\n"+ "CACHEFILE=" + targetLocalRemoteFolderPath + File.separator + ".cloudsync.cache";
        config += "\n"+ "LOGFILE=" + targetLocalRemoteFolderPath + File.separator + ".cloudsync.log";
        Files.write(configFile.toPath(), config.getBytes(), StandardOpenOption.CREATE);
                
        System.out.println("RootFolder: " + rootFolder.getAbsolutePath());
        System.out.println("TargetFolder: " + targetLocalRemoteFolder.getAbsolutePath());

        
        String[] args = new String[]{
            "--nocache",
            "--backup",
            rootFolder.getAbsolutePath(),
            "--name",
            "Test1Backup",
            "--config",
            configFile.getAbsolutePath()
        };
        Cloudsync.main(args);
        
        
        restoreFolder.mkdir();
        args = new String[]{
            "--nocache",
            "--restore",
            restoreFolder.getAbsolutePath(),
            "--name",
            "Test1Backup",
            "--config",
            configFile.getAbsolutePath()
        };
        Cloudsync.main(args);
        
        
        assertTrue(hierarchieEquals(rootFolder, restoreFolder));
        FileUtils.deleteDirectory(rootFolder);
    }
    
    
    /**
     * Create a specific hierarchy
     * Modify files
     * Check if file is in history.
     */
    @Test
    public void test2() throws IOException, ParseException, CloudsyncException, FileIOException, UsageException, InfoException {
        File rootFolder = Files.createTempDirectory("srcTest2").toFile();
        File targetLocalRemoteFolder = Files.createTempDirectory("targetRemoteFolder").toFile();
        File restoreFolder = new File(rootFolder.getParent(),rootFolder.getName()+"_restore");
        File configFile = Files.createTempFile("test2Config", ".config").toFile();
        
        System.out.println("Test2");
        System.out.println("RootFolder: " + rootFolder.getAbsolutePath());
        System.out.println("TargetFolder: " + targetLocalRemoteFolder.getAbsolutePath());

        String targetLocalRemoteFolderPath = FilesystemHelper.fixPathSeparators(targetLocalRemoteFolder.getAbsolutePath());

        String config = "REMOTE_CONNECTOR=LocalFilesystem";
        config += "\n"+ "PASSPHRASE=1234567";
        config += "\n"+ "TARGET_DIR=" + targetLocalRemoteFolderPath;
        config += "\n"+ "CACHEFILE=" + targetLocalRemoteFolderPath + File.separator + ".cloudsync.cache";
        config += "\n"+ "LOGFILE=" + targetLocalRemoteFolderPath + File.separator + ".cloudsync.log";
        config += "\n"+ "HISTORY=3";
        Files.write(configFile.toPath(), config.getBytes(), StandardOpenOption.CREATE);
           
        /**
         * rootFolder
         *  /file1
         *  /subfolder1
         *      /file2
         *      /subfolder2
         *          /file3
         */
        File file1 = new File(rootFolder,"file1");
        File subfolder1 = new File(rootFolder,"subfolder1");
        File file2 = new File(subfolder1,"file2");
        File subfolder2 = new File(subfolder1,"subfolder2");
        File file3 = new File(subfolder2,"file3");
        subfolder1.mkdirs();
        subfolder2.mkdirs();
        
        Files.write(file1.toPath(), "file 1 content".getBytes("UTF-8"), StandardOpenOption.CREATE_NEW);
        Files.write(file2.toPath(), "file 2 content".getBytes("UTF-8"), StandardOpenOption.CREATE_NEW);
        Files.write(file3.toPath(), "file 3 content".getBytes("UTF-8"), StandardOpenOption.CREATE_NEW);
        
        String[] args = new String[]{
            "--nocache",
            "--backup",
            rootFolder.getAbsolutePath(),
            "--name",
            "Test2Backup",
            "--config",
            configFile.getAbsolutePath()
        };
        Cloudsync.main(args);

        // Update files content
        Files.write(file1.toPath(), "new file 1 content updated".getBytes("UTF-8"), StandardOpenOption.TRUNCATE_EXISTING);
        Files.write(file2.toPath(), "new file 2 content updated".getBytes("UTF-8"), StandardOpenOption.TRUNCATE_EXISTING);
        Files.write(file3.toPath(), "new file 3 content updated".getBytes("UTF-8"), StandardOpenOption.TRUNCATE_EXISTING);
        Cloudsync.main(args);
        
        // Check if history exist for updated files
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy.MM.dd");
        File historyFolder = new File(targetLocalRemoteFolder,"Test2Backup"+"_history_"+sdf.format(new Date()));
        
        CmdOptions cmdOptions = new CmdOptions(args);
        cmdOptions.parse();
        Crypt crypt = new Crypt(cmdOptions);
        assertEquals(historyFolder.listFiles().length,3);
        for(File f1 : historyFolder.listFiles()) {
            if(f1.getName().endsWith(".metadata")) {
                //TODO
            }
            else if(f1.isFile()) {
                assertEquals(crypt.decryptText(f1.getName()), "file1");
                _assertEncryptedFileContentEquals(f1,crypt,"file 1 content");
            }
            else {
                assertEquals(crypt.decryptText(f1.getName()), "subfolder1");
                assertEquals(f1.listFiles().length,3);
                for(File f2 : f1.listFiles()) {
                    if(f2.getName().endsWith(".metadata")) {
                        //TODO
                    }
                    else if(f2.isFile()) {
                        assertEquals(crypt.decryptText(f2.getName()), "file2");
                        _assertEncryptedFileContentEquals(f2,crypt,"file 2 content");
                    }
                    else {
                        assertEquals(f2.listFiles().length,2);
                        assertEquals(crypt.decryptText(f2.getName()), "subfolder2");
                        for(File f3 : f2.listFiles()) {
                            if(f3.getName().endsWith(".metadata")) {
                                //TODO
                            }
                            else {
                                assertEquals(crypt.decryptText(f3.getName()), "file3");
                                _assertEncryptedFileContentEquals(f3,crypt,"file 3 content");
                            }
                        }
                    }
                }
            }
        }     
        
        restoreFolder.mkdir();
        args = new String[]{
            "--nocache",
            "--restore",
            restoreFolder.getAbsolutePath(),
            "--name",
            "Test2Backup",
            "--config",
            configFile.getAbsolutePath()
        };
        Cloudsync.main(args);
        
        
        assertTrue(hierarchieEquals(rootFolder, restoreFolder));
        FileUtils.deleteDirectory(rootFolder);
    }
   
    private void _assertEncryptedFileContentEquals(File file, Crypt crypt, String content) throws IOException, CloudsyncException {
        byte[] encryptedFileContent = Files.readAllBytes(file.toPath());
        ByteArrayInputStream bais = new ByteArrayInputStream(encryptedFileContent);
        byte[] fileContent = IOUtils.toByteArray(crypt.decryptData(bais));
        String s = new String(fileContent, "UTF-8");
        assertEquals(s,content);
    }
    
    
    public static File createRandomHierarchie() throws IOException {
        Random random = new Random();

        int prefixLength = random.nextInt(10)+1;
        String prefix = "src"+org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric(prefixLength);

        File root = Files.createTempDirectory(prefix).toFile();
        root.mkdirs();
        createRandomHierarchie(root,(random.nextDouble()+1.)*3.);
        return root;
    }

    public static void createRandomHierarchie(File parent, double nbFolderProba) throws IOException {
        Random random = new Random();

        double r = random.nextDouble();
        while(r < nbFolderProba) {
            nbFolderProba = nbFolderProba / (random.nextDouble() + 1.);
            File folder = createRandomFolder(parent,random.nextInt(20));
            createRandomHierarchie(folder,nbFolderProba);
            r = random.nextDouble();
        }
    }
    
    
    
    public static void createRandomFile(File inFolder) throws IOException {
        Random random = new Random();
        
        int prefixLength = random.nextInt(10)+3;
        String prefix = org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric(prefixLength);
        String suffix = org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric(3);
        File f = File.createTempFile(prefix, suffix, inFolder);
        
        
        int n = random.nextInt(50);
        while(n != 0) {
            int l = random.nextInt(1000);
            byte[] data = new byte[l];
            random.nextBytes(data);
            Files.write(f.toPath(), data, StandardOpenOption.CREATE,StandardOpenOption.APPEND);
            n = n - 1;
        }
    }

    
    public static File createRandomFolder(File inFolder, int nbFile) throws IOException {
        Random random = new Random();
        int prefixLength = random.nextInt(10)+1;
        String prefix = org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric(prefixLength);
        File result = Files.createTempDirectory(inFolder.toPath(), prefix).toFile();
        result.mkdirs();
        for(int i = 0;i<nbFile;i++) {
            createRandomFile(result);
        }
        return result;
    }
    
    
    public static boolean hierarchieEquals(File root1,File root2) throws IOException {
        File[] files1 = root1.listFiles();
        File[] files2 = root2.listFiles();
        if(files1.length != files2.length) {
            return false;
        }
        
        Map<String,File> mapFiles2 = new HashMap<>();
        for(File f : files2) {
            mapFiles2.put(f.getName(), f);
        }
        
        for(File f1 : files1) {
            if(!mapFiles2.containsKey(f1.getName())) {
                return false;
            }
            
            File f2 = mapFiles2.get(f1.getName());
            if(f1.isDirectory() && f2.isDirectory()) {
                if(!hierarchieEquals(f1,f2)) {
                    return false;
                }
            }
            else if(f1.isFile() && f2.isFile()) {
                if(!FileUtils.contentEquals(f1, f2)) {
                    return false;
                }
            }
            else {
                return false;
            }
            
            mapFiles2.remove(f1.getName());
        }
        
        if(!mapFiles2.isEmpty()) {
            return false;
        }
        
        return true;
    }
    
}