How to clone an Apache file directory from a website?  - Page 1
Forum section: electronics / general

author: Jackster
author_rank: Frequent Contributor
timestamp: December 15, 2022, 11:27:39 pm
content: Doing a bit of file archival work and we have a few public-facing Apache file directories like in this screenshot.The directory structure is pretty simple but there are many of them.The files are pretty standard but some are text files and some are binary.What I would like to do is basically clone those directories onto my local storage.I have tried WGET so far with some success.Code:wget -m -p -E -k -K -np "http://domain.ltd/dir/"As well as an assortment of flags.But I am finding that it does not download the txt files correctly and sometimes the binary files are downloaded incorrectly also.Does anyone have a better solution?I don't need to generate the index pages either.

author: Jackster
author_rank: Frequent Contributor
timestamp: December 15, 2022, 11:41:36 pm
content: A few solutions have poped up such ashttps://github.com/ArchiveBox/ArchiveBoxand Archive Team's self hosted Warrior.

author: retiredfeline
author_rank: Frequent Contributor
timestamp: December 15, 2022, 11:48:40 pm
content: It's been years since I had to do anything like this but I think I used httrack.

author: Jackster
author_rank: Frequent Contributor
timestamp: December 16, 2022, 12:07:06 am
content: Quote from: retiredfeline on December 15, 2022, 11:48:40 pmThat is what I used first. Tried and tested bit of software. But it also had issues with the binary files and text files.

author: Whales
author_rank: Super Contributor
timestamp: December 16, 2022, 12:43:06 am
content: What exactly are the issues with the files you have downloaded? Please give as much detail as you can.Are you trying to clone that site's code assets including .php files so you can rehost it yourself? You probably can't do that from a web browser, the Apache instance will likely "run" the php pages when you try to access them (rather than serve you the raw file contents). You will need a different form of access to the server (eg a shared hosting control panel or ssh/scp/sftp).

author: retiredfeline
author_rank: Frequent Contributor
timestamp: December 16, 2022, 12:45:03 am
content: What sorts of issues? Linux doesn't have attributes for binary or text files but you can configure the Apache server your end to associate MIME types with particular file extensions. This is a configuration setup on the origin server that isn't exported to the world.I noticed now that it looks like a Wordpress site. Dynamic sites like that where content is programmatically generated cannot be cloned from the website. You need access to the underlying Wordpress installation.

author: AndyBeez
author_rank: Frequent Contributor
timestamp: December 16, 2022, 12:59:30 am
content: Gain root access and FTP / SSH all of the files into a zip or tar archive.

author: Jackster
author_rank: Frequent Contributor
timestamp: December 16, 2022, 02:31:36 am
content: Should have added more info..It is a publicly accessible directory for a game's downloadable mods and maps.The files are compiled data files and txt configs.I don't have access to the actual server's backend.

author: golden_labels
author_rank: Super Contributor
timestamp: December 16, 2022, 04:09:21 am
content: Quote from: Jackster on December 16, 2022, 02:31:36 amYou still did not answerWhale’s questionabout the problems you are experiencing. We do not know, what you did download. Neither we do know what you consider “incorrect” and — most importantly — what is your reasoning behind considering something “incorrect”.On a separate note: while dumping websites please consider using wget’s--limit-rate,--waitand--random-wait. That will decrease load variance, making it affect normal users less.

author: Nominal Animal
author_rank: Super Contributor
timestamp: December 16, 2022, 12:43:12 pm
content: The only problem I can think of wrt. plain text files is that the server supplies aContent-Type: text/plain; charset=encodingwith differentencodingthan you use on your local machine. (Typically, the server providesUTF-8, and you use e.g.Windows Western Europeanakawin-1252aka Codepage 1252.)The effect is that all ASCII characters are fine, but non-ascii characters are mangled.One other quirk is that you might use a text editor that requires Windows CR-LF newline convention, whereas Linux, BSDs, Macs use LF-only.There is no "fix" nor "error".What you need to do, is view the file in your browser, and change the encoding to what the server used.(Browser developers are stupid, and don't have any way of setting local text files' encoding toUTF-8, "because it is not a legacy encoding and is therefore unsuitable as a default". Yeah, it is one of my pet peeves.)Because you intend to do an archival, I recommend against recoding the files with e.g.iconv.

author: Jackster
author_rank: Frequent Contributor
timestamp: December 16, 2022, 01:34:29 pm
content: Thanks to a suggestion by someone on ArchiveTeam's IRC, OpenDirectoryDownloader to index the URLs and then use WGET to download all the files.It is working well and not messing up the config files.https://github.com/KoalaBear84/OpenDirectoryDownloader