Repository: joke2k/faker

Issue:
Feature Request: Fake Binary Content
Let's say I want to write unit tests for my web service that has a file upload feature.
It would be great if I can fake random binary content of popular binary file types, especially images and documents, like:

- *.jpg
- *.pdf
- *.doc/docx

I think image uploading is probably the most common type of uploaded binary content, so it may be a good place to start.

**Why would someone need this?**
This could be great mainly for testing file upload APIs for several reasons.
It's way easier if the unit tests can generate fake binary content, so they will not depend on actual fixed binary files (and syncing them between teammates, YUCK).

**Some use case examples:**
- A user uploads any format of an image (jpg, png, bmp, etc.) and the server needs to recognize the image type (by its binary structure, headers etc.) and store/convert/compress/whatever according to it.
- A user uploads a document in a popular format (.pdf, .docx, .pages) and the server needs to read the content for parsing/analysis/whatever.


Discussion:
There's a couple of code samples I found to get an idea of how we can accomplish this for **images**.
These depend on [Pillow ](https://pypi.python.org/pypi/Pillow/3.0.0)(a fork of [Python Image Library](https://pypi.python.org/pypi/PIL) which has better support and an easier API).

- [random-image](https://github.com/tephyr/random-image)
- some [stackoverflow answer](https://stackoverflow.com/a/10901092/3134477)

@fcurella I will try and take a swing at this, but my main concern are the additional dependencies. Do we keep dependencies like Pillow optional?
I am already done with fake zip and tar files, both with options to specify the number of files, the minimum file size per file, the total uncompressed size, and compression type to use. Hopefully you can give me feedback regarding additional dependencies @fcurella.
@malefice Is there a PR for this?
@mabuelhagag nope, it is still in my local branch. My other PR #1052 has some significant changes, and I would rather have that merged first, so that my PR for this will already be rebased properly.

I think it has been a busy last couple of days in the US because it is almost Thanksgiving and Black Friday, so I suggest checking back later.

Resolving patch:
diff --git a/faker/providers/misc/__init__.py b/faker/providers/misc/__init__.py
--- a/faker/providers/misc/__init__.py
+++ b/faker/providers/misc/__init__.py
@@ -2,9 +2,14 @@
 
 from __future__ import unicode_literals
 import hashlib
+import io
 import string
+import tarfile
 import uuid
 import sys
+import zipfile
+
+import six
 
 from .. import BaseProvider
 
@@ -122,3 +127,119 @@ def password(
             chars[index] = required_tokens[i]
 
         return ''.join(chars)
+
+    def zip(self, uncompressed_size=65536, num_files=1, min_file_size=4096, compression=None):
+        """
+        Returns a random valid zip archive
+
+        :param uncompressed_size: Total size of files before compression, 16 KiB by default
+        :param num_files: Number of files archived in resulting zip file, 1 file by default
+        :param min_file_size: Minimum size of each file before compression, 4 KiB by default
+        :param compression: bzip2 or bz2 for BZIP2, lzma or xz for LZMA,
+                             deflate gzip or gz for GZIP, otherwise no compression
+
+        :return: Bytes object representation of zip archive
+        """
+
+        if any([
+            not isinstance(num_files, int) or num_files <= 0,
+            not isinstance(min_file_size, int) or min_file_size <= 0,
+            not isinstance(uncompressed_size, int) or uncompressed_size <= 0,
+        ]):
+            raise ValueError(
+                '`num_files`, `min_file_size`, and `uncompressed_size` must be positive integers',
+            )
+        if min_file_size * num_files > uncompressed_size:
+            raise AssertionError(
+                '`uncompressed_size` is smaller than the calculated minimum required size',
+            )
+        if compression in ['bzip2', 'bz2']:
+            if six.PY2:
+                raise RuntimeError('BZIP2 compression is not supported in Python 2')
+            compression = zipfile.ZIP_BZIP2
+        elif compression in ['lzma', 'xz']:
+            if six.PY2:
+                raise RuntimeError('LZMA compression is not supported in Python 2')
+            compression = zipfile.ZIP_LZMA
+        elif compression in ['deflate', 'gzip', 'gz']:
+            compression = zipfile.ZIP_DEFLATED
+        else:
+            compression = zipfile.ZIP_STORED
+
+        zip_buffer = io.BytesIO()
+        remaining_size = uncompressed_size
+        with zipfile.ZipFile(zip_buffer, mode='w', compression=compression) as zip_handle:
+            for file_number in range(1, num_files + 1):
+                filename = self.generator.pystr() + str(file_number)
+
+                max_allowed_size = remaining_size - (num_files - file_number) * min_file_size
+                if file_number < num_files:
+                    file_size = self.generator.random.randint(min_file_size, max_allowed_size)
+                    remaining_size = remaining_size - file_size
+                else:
+                    file_size = remaining_size
+
+                data = self.generator.binary(file_size)
+                if six.PY3:
+                    zip_handle.writestr(filename, data)
+                else:
+                    zip_handle.writestr(filename, str(data))
+        return zip_buffer.getvalue()
+
+    def tar(self, uncompressed_size=65536, num_files=1, min_file_size=4096, compression=None):
+        """
+        Returns a random valid tar
+
+        :param uncompressed_size: Total size of files before compression, 16 KiB by default
+        :param num_files: Number of files archived in resulting zip file, 1 file by default
+        :param min_file_size: Minimum size of each file before compression, 4 KiB by default
+        :param compression: gzip or gz for GZIP, bzip2 or bz2 for BZIP2,
+                            lzma or xz for LZMA, otherwise no compression
+
+        :return: Bytes object representation of tar
+        """
+        if any([
+            not isinstance(num_files, int) or num_files <= 0,
+            not isinstance(min_file_size, int) or min_file_size <= 0,
+            not isinstance(uncompressed_size, int) or uncompressed_size <= 0,
+        ]):
+            raise ValueError(
+                '`num_files`, `min_file_size`, and `uncompressed_size` must be positive integers',
+            )
+        if min_file_size * num_files > uncompressed_size:
+            raise AssertionError(
+                '`uncompressed_size` is smaller than the calculated minimum required size',
+            )
+        if compression in ['gzip', 'gz']:
+            mode = 'w:gz'
+        elif compression in ['bzip2', 'bz2']:
+            mode = 'w:bz2'
+        elif compression in ['lzma', 'xz']:
+            if six.PY2:
+                raise RuntimeError('LZMA compression is not supported in Python 2')
+            mode = 'w:xz'
+        else:
+            mode = 'w'
+
+        tar_buffer = io.BytesIO()
+        remaining_size = uncompressed_size
+        with tarfile.open(mode=mode, fileobj=tar_buffer) as tar_handle:
+            for file_number in range(1, num_files + 1):
+                file_buffer = io.BytesIO()
+                filename = self.generator.pystr() + str(file_number)
+
+                max_allowed_size = remaining_size - (num_files - file_number) * min_file_size
+                if file_number < num_files:
+                    file_size = self.generator.random.randint(min_file_size, max_allowed_size)
+                    remaining_size = remaining_size - file_size
+                else:
+                    file_size = remaining_size
+
+                tarinfo = tarfile.TarInfo(name=filename)
+                data = self.generator.binary(file_size)
+                file_buffer.write(data)
+                tarinfo.size = len(file_buffer.getvalue())
+                file_buffer.seek(0)
+                tar_handle.addfile(tarinfo, file_buffer)
+                file_buffer.close()
+        return tar_buffer.getvalue()


Test patch:
diff --git a/tests/providers/test_misc.py b/tests/providers/test_misc.py
--- a/tests/providers/test_misc.py
+++ b/tests/providers/test_misc.py
@@ -1,5 +1,11 @@
+# coding=utf-8
+
+from __future__ import unicode_literals
+import io
+import tarfile
 import unittest
 import uuid
+import zipfile
 import six
 
 from faker import Faker
@@ -36,3 +42,315 @@ def test_uuid4_seedability(self):
             Faker.seed(random_seed)
             new_uuids = [new_fake.uuid4() for i in range(1000)]
             assert new_uuids == expected_uuids
+
+    def test_zip_invalid_file(self):
+        with self.assertRaises(ValueError):
+            self.factory.zip(num_files='1')
+
+        with self.assertRaises(ValueError):
+            self.factory.zip(num_files=0)
+
+        with self.assertRaises(ValueError):
+            self.factory.zip(min_file_size='1')
+
+        with self.assertRaises(ValueError):
+            self.factory.zip(min_file_size=0)
+
+        with self.assertRaises(ValueError):
+            self.factory.zip(uncompressed_size='1')
+
+        with self.assertRaises(ValueError):
+            self.factory.zip(uncompressed_size=0)
+
+    def test_zip_one_byte_undersized(self):
+        Faker.seed(0)
+        for _ in range(10):
+            num_files = self.factory.random.randint(1, 100)
+            min_file_size = self.factory.random.randint(1, 1024)
+            uncompressed_size = num_files * min_file_size - 1
+
+            # Will always fail because of bad size requirements
+            with self.assertRaises(AssertionError):
+                self.factory.zip(
+                    uncompressed_size=uncompressed_size, num_files=num_files,
+                    min_file_size=min_file_size,
+                )
+
+    def test_zip_exact_minimum_size(self):
+        Faker.seed(0)
+        for _ in range(10):
+            num_files = self.factory.random.randint(1, 100)
+            min_file_size = self.factory.random.randint(1, 1024)
+            uncompressed_size = num_files * min_file_size
+
+            zip_bytes = self.factory.zip(
+                uncompressed_size=uncompressed_size, num_files=num_files,
+                min_file_size=min_file_size,
+            )
+            zip_buffer = io.BytesIO(zip_bytes)
+            with zipfile.ZipFile(zip_buffer, 'r') as zip_handle:
+                # Verify zip archive is good
+                assert zip_handle.testzip() is None
+
+                # Verify zip archive has the correct number of files
+                infolist = zip_handle.infolist()
+                assert len(infolist) == num_files
+
+                # Every file's size will be the minimum specified
+                total_size = 0
+                for info in infolist:
+                    assert info.file_size == min_file_size
+                    total_size += info.file_size
+
+                # The file sizes should sum up to the specified uncompressed size
+                assert total_size == uncompressed_size
+
+    def test_zip_over_minimum_size(self):
+        Faker.seed(0)
+        for _ in range(10):
+            num_files = self.factory.random.randint(1, 100)
+            min_file_size = self.factory.random.randint(1, 1024)
+            expected_extra_bytes = self.factory.random.randint(1, 1024 * 1024)
+            uncompressed_size = num_files * min_file_size + expected_extra_bytes
+
+            zip_bytes = self.factory.zip(
+                uncompressed_size=uncompressed_size, num_files=num_files,
+                min_file_size=min_file_size,
+            )
+            zip_buffer = io.BytesIO(zip_bytes)
+            with zipfile.ZipFile(zip_buffer, 'r') as zip_handle:
+                # Verify zip archive is good
+                assert zip_handle.testzip() is None
+
+                # Verify zip archive has the correct number of files
+                infolist = zip_handle.infolist()
+                assert len(infolist) == num_files
+
+                # Every file's size will be at least the minimum specified
+                extra_bytes = 0
+                total_size = 0
+                for info in infolist:
+                    assert info.file_size >= min_file_size
+                    total_size += info.file_size
+                    if info.file_size > min_file_size:
+                        extra_bytes += (info.file_size - min_file_size)
+
+                # The file sizes should sum up to the specified uncompressed size
+                # and the extra bytes counted must be equal to the one expected
+                assert total_size == uncompressed_size
+                assert extra_bytes == expected_extra_bytes
+
+    @unittest.skipIf(six.PY2, 'Python 3 only')
+    def test_zip_compression_py3(self):
+        Faker.seed(0)
+        num_files = 10
+        min_file_size = 512
+        uncompressed_size = 50 * 1024
+        compression_mapping = [
+            ('deflate', zipfile.ZIP_DEFLATED),
+            ('gzip', zipfile.ZIP_DEFLATED),
+            ('gz', zipfile.ZIP_DEFLATED),
+            ('bzip2', zipfile.ZIP_BZIP2),
+            ('bz2', zipfile.ZIP_BZIP2),
+            ('lzma', zipfile.ZIP_LZMA),
+            ('xz', zipfile.ZIP_LZMA),
+            (None, zipfile.ZIP_STORED),
+        ]
+        for compression, compress_type in compression_mapping:
+            zip_bytes = self.factory.zip(
+                uncompressed_size=uncompressed_size, num_files=num_files,
+                min_file_size=min_file_size, compression=compression,
+            )
+            zip_buffer = io.BytesIO(zip_bytes)
+            with zipfile.ZipFile(zip_buffer, 'r') as zip_handle:
+                # Verify zip archive is good
+                assert zip_handle.testzip() is None
+
+                # Verify compression type used
+                for info in zip_handle.infolist():
+                    assert info.compress_type == compress_type
+
+    @unittest.skipIf(six.PY3, 'Python 2 only')
+    def test_zip_compression_py2(self):
+        Faker.seed(0)
+        num_files = 10
+        min_file_size = 512
+        uncompressed_size = 50 * 1024
+        compression_mapping = [
+            ('deflate', zipfile.ZIP_DEFLATED),
+            ('gzip', zipfile.ZIP_DEFLATED),
+            ('gz', zipfile.ZIP_DEFLATED),
+            (None, zipfile.ZIP_STORED),
+        ]
+        for compression, compress_type in compression_mapping:
+            zip_bytes = self.factory.zip(
+                uncompressed_size=uncompressed_size, num_files=num_files,
+                min_file_size=min_file_size, compression=compression,
+            )
+            zip_buffer = io.BytesIO(zip_bytes)
+            with zipfile.ZipFile(zip_buffer, 'r') as zip_handle:
+                # Verify zip archive is good
+                assert zip_handle.testzip() is None
+
+                # Verify compression type used
+                for info in zip_handle.infolist():
+                    assert info.compress_type == compress_type
+
+        # BZIP2 and LZMA are not supported in Python 2
+        for compression in ['bzip2', 'bz2', 'lzma', 'xz']:
+            with self.assertRaises(RuntimeError):
+                self.factory.zip(
+                    uncompressed_size=uncompressed_size, num_files=num_files,
+                    min_file_size=min_file_size, compression=compression,
+                )
+
+    def test_tar_invalid_file(self):
+        with self.assertRaises(ValueError):
+            self.factory.tar(num_files='1')
+
+        with self.assertRaises(ValueError):
+            self.factory.tar(num_files=0)
+
+        with self.assertRaises(ValueError):
+            self.factory.tar(min_file_size='1')
+
+        with self.assertRaises(ValueError):
+            self.factory.tar(min_file_size=0)
+
+        with self.assertRaises(ValueError):
+            self.factory.tar(uncompressed_size='1')
+
+        with self.assertRaises(ValueError):
+            self.factory.tar(uncompressed_size=0)
+
+    def test_tar_one_byte_undersized(self):
+        Faker.seed(0)
+        for _ in range(10):
+            num_files = self.factory.random.randint(1, 100)
+            min_file_size = self.factory.random.randint(1, 1024)
+            uncompressed_size = num_files * min_file_size - 1
+
+            # Will always fail because of conflicting size requirements
+            with self.assertRaises(AssertionError):
+                self.factory.tar(
+                    uncompressed_size=uncompressed_size, num_files=num_files,
+                    min_file_size=min_file_size,
+                )
+
+    def test_tar_exact_minimum_size(self):
+        Faker.seed(0)
+        for _ in range(10):
+            num_files = self.factory.random.randint(1, 100)
+            min_file_size = self.factory.random.randint(1, 1024)
+            uncompressed_size = num_files * min_file_size
+
+            tar_bytes = self.factory.tar(
+                uncompressed_size=uncompressed_size, num_files=num_files,
+                min_file_size=min_file_size,
+            )
+            tar_buffer = io.BytesIO(tar_bytes)
+            with tarfile.open(fileobj=tar_buffer) as tar_handle:
+                # Verify tar has the correct number of files
+                members = tar_handle.getmembers()
+                assert len(members) == num_files
+
+                # Every file's size will be the minimum specified
+                total_size = 0
+                for member in members:
+                    assert member.size == min_file_size
+                    total_size += member.size
+
+                # The file sizes should sum up to the specified uncompressed size
+                assert total_size == uncompressed_size
+
+    def test_tar_over_minimum_size(self):
+        Faker.seed(0)
+        for _ in range(10):
+            num_files = self.factory.random.randint(1, 100)
+            min_file_size = self.factory.random.randint(1, 1024)
+            expected_extra_bytes = self.factory.random.randint(1, 1024 * 1024)
+            uncompressed_size = num_files * min_file_size + expected_extra_bytes
+
+            tar_bytes = self.factory.tar(
+                uncompressed_size=uncompressed_size, num_files=num_files,
+                min_file_size=min_file_size,
+            )
+            tar_buffer = io.BytesIO(tar_bytes)
+            with tarfile.open(fileobj=tar_buffer) as tar_handle:
+                # Verify tar has the correct number of files
+                members = tar_handle.getmembers()
+                assert len(members) == num_files
+
+                # Every file's size will be at least the minimum specified
+                extra_bytes = 0
+                total_size = 0
+                for member in members:
+                    assert member.size >= min_file_size
+                    total_size += member.size
+                    if member.size > min_file_size:
+                        extra_bytes += (member.size - min_file_size)
+
+                # The file sizes should sum up to the specified uncompressed size
+                # and the extra bytes counted should be the one we expect
+                assert total_size == uncompressed_size
+                assert extra_bytes == expected_extra_bytes
+
+    @unittest.skipIf(six.PY2, 'Python 3 only')
+    def test_tar_compression_py3(self):
+        Faker.seed(0)
+        num_files = 25
+        min_file_size = 512
+        uncompressed_size = 50 * 1024
+        compression_mapping = [
+            ('gzip', 'r:gz'),
+            ('gz', 'r:gz'),
+            ('bzip2', 'r:bz2'),
+            ('bz2', 'r:bz2'),
+            ('lzma', 'r:xz'),
+            ('xz', 'r:xz'),
+            (None, 'r'),
+        ]
+
+        for compression, read_mode in compression_mapping:
+            tar_bytes = self.factory.tar(
+                uncompressed_size=uncompressed_size, num_files=num_files,
+                min_file_size=min_file_size, compression=compression,
+            )
+            tar_buffer = io.BytesIO(tar_bytes)
+            with tarfile.open(fileobj=tar_buffer, mode=read_mode) as tar_handle:
+                # Verify tar has the correct number of files
+                members = tar_handle.getmembers()
+                assert len(members) == num_files
+
+    @unittest.skipIf(six.PY3, 'Python 2 only')
+    def test_tar_compression_py2(self):
+        Faker.seed(0)
+        num_files = 25
+        min_file_size = 512
+        uncompressed_size = 50 * 1024
+        compression_mapping = [
+            ('gzip', 'r:gz'),
+            ('gz', 'r:gz'),
+            ('bzip2', 'r:bz2'),
+            ('bz2', 'r:bz2'),
+            (None, 'r'),
+        ]
+
+        for compression, read_mode in compression_mapping:
+            tar_bytes = self.factory.tar(
+                uncompressed_size=uncompressed_size, num_files=num_files,
+                min_file_size=min_file_size, compression=compression,
+            )
+            tar_buffer = io.BytesIO(tar_bytes)
+            with tarfile.open(fileobj=tar_buffer, mode=read_mode) as tar_handle:
+                # Verify tar has the correct number of files
+                members = tar_handle.getmembers()
+                assert len(members) == num_files
+
+        # LZMA is not supported in Python 2
+        for compression in ['lzma', 'xz']:
+            with self.assertRaises(RuntimeError):
+                self.factory.tar(
+                    uncompressed_size=uncompressed_size, num_files=num_files,
+                    min_file_size=min_file_size, compression=compression,
+                )