Package: dj-bgfiles
Generate files in the background and allow users to pick them up afterwards using a token.DocumentationThe “full” documentation is athttps://dj-bgfiles.readthedocs.org.QuickstartFirst ensure you’re using Django 1.8.4 or up. Next, install dj-bgfiles:pip install dj-bgfilesAdd it yourINSTALLED_APPSand make sure to run themigratecommand to create the necessary tables.FeaturesThe most common use case is delivering sizeable data exports or complex reports to users without overloading the web
server, which is whybgfilesprovides tools to:Store a file generation/download requestProcess that request later on, e.g. by using Celery or a cron jobSend a notification to the requester including a secure link to download the fileAnd serving the file to the requester (or others if applicable)ExampleIn our hypothetical project we want to provide users with the option to generate an xlsx file containing login events. Here’s our model:class LoginEvent(models.Model):
    user = models.ForeignKey(User, blank=True, null=True)
    succeeded = models.BooleanField(default=True)
    created_at = models.DateTimeField(auto_now_add=True)

    def save(self, *args, **kwargs):
        self.succeeded = True if self.user_id else False
        super(LoginEvent, self).save(*args, **kwargs)Our export allows a user to specify a timeframe through a form:class LoginExportFilterForm(models.Model):
    from_date = models.DateField()
    until_date = models.DateField(required=False)

    def apply_filters(self, queryset):
        """Apply the filters to our initial queryset."""
        data = self.cleaned_data
        queryset = queryset.filter(created_at__gte=data['from_date'])
        until_date = data.get('until_date')
        if until_date:
            queryset = queryset.filter(created_at__lte=until_date)
        return querysetSo our view would look like this initially:def export_login_events(request):
    if request.method == 'POST':
        form = LoginExportFilterForm(data=request.POST)
        if form.is_valid():
            # Grab our events
            events = LoginEvent.objects.all()
            # Apply the filters the user supplied
            events = form.apply_filters(events)
            # And write everything to an xlsx file and serve it as a HttpResponse
            return write_xlsx(events)
    else:
        form = LoginExportFilterForm()
    return render(request, 'reports/login_events_filter.html', context={'form': form})But overnight, as it happens, our app got really popular so we have a ton of login events. We want to offload the
creation of large files to a background process that’ll send a notification to the user when the file’s ready with a
link to download it.Enterbgfiles.Manually, using a Celery taskHere’s how we could handle this manually using ourtoolboxand a Celery task:from bgfiles import toolbox

def export_login_events(request):
    if request.method == 'POST':
        form = LoginExportFilterForm(data=request.POST)
        if form.is_valid():
            # Grab our events
            events = LoginEvent.objects.all()
            # Apply the filters the user supplied
            events = form.apply_filters(events)
            # We want to limit online file creation to at most 10000 events
            max_nr_events = 10000
            nr_events = events.count()
            if nr_events <= max_nr_events:
                # Ok, deliver our file to the user
                return write_xlsx(events)
            # The file would be too big. So let's grab our criteria
            criteria = request.POST.urlencode()
            # The content type is optional
            content_type = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
            # Now add a file request
            with transaction.atomic():
                file_request = toolbox.add_request(criteria, file_type='login_events',
                                                   requester=request.user, content_type=content_type)
            # Schedule a Celery task
            export_login_events_task.delay(file_request.id)
            # And let the user know we'll get back to them on that
            context = {'nr_events': nr_events, 'max_nr_events': max_nr_events}
            return render(request, 'reports/delayed_response.html', context=context)

    else:
        form = LoginExportFilterForm()
    return render(request, 'reports/login_events_filter.html', context={'form': form})When we add a file request, we first marshall our criteria to something our database can store and we can easily
unmarshall later on. We specify a file type to support requests for different types of files and also record the
user that requested the file. The defaultbgfileslogic assumes only the user performing the request should be
able to download the file later on.Anyway, our Celery task to create the file in the background might look like this:from bgfiles.models import FileRequest
from django.http import QueryDict

@task
def export_login_events_task(file_request_id):
    # Grab our request. You might want to lock it or check if it's already been processed in the meanwhile.
    request = FileRequest.objects.get(id=file_request_id)
    # Restore our criteria
    criteria = QueryDict(request.criteria)
    # Build our form and apply the filters as specified by the criteria
    form = LoginExportFilterForm(data=criteria)
    events = LoginEvent.objects.all()
    events = form.apply_filters(events)
    # Write the events to an xlsx buffer (simplified)
    contents = write_xlsx(events)
    # Attach the contents to the request and add a filename
    toolbox.attach_file(request, contents, filename='login_events.xlsx'):
    # Generate a token for the requester of the file
    token = toolbox.create_token(request)
    # Grab the download url including our token
    download_url = reverse('mydownloads:serve', kwargs={'token': token})
    # And send out an email containing the link
    notifications.send_file_ready(request.requester, download_url, request.filename)It should be pretty obvious what the above code is doing. Note that restoring our criteria is easy: we simply
instantiate a QueryDict. Yes, there’s a bit of code duplication. We’ll get to that later on.Manually, using a cron jobLet’s defer the generation to a cron job that will send out an email to our user. Our view would look the same, except
we won’t schedule a Celery task. Our cron logic then might look like this:from bgfiles import toolbox
from bgfiles.models import FileRequest
from django.http import QueryDict

def process_file_requests():
    # Only grab the requests that still need to be processed
    requests = FileRequest.objects.to_handle()
    # Process each one by delegating to a specific handler
    for request in requests:
        if request.file_type == 'login_events':
            process_login_events(request)
        elif request.file_type == 'something_else:
            process_something_else(request)
        else:
            raise Exception('Unsupported file type %s' % request.file_type)

def process_login_events(request):
    # Restore our criteria
    criteria = QueryDict(request.criteria)
    # Build our form and apply the filters as specified by the criteria
    form = LoginExportFilterForm(data=criteria)
    events = LoginEvent.objects.all()
    events = form.apply_filters(events)
    # Write the events to an xlsx buffer (simplified)
    contents = write_xlsx(events)
    # Attach the contents to the request and add a filename
    toolbox.attach_file(request, contents, filename='login_events.xlsx'):
    # Generate a token for the requester of the file
    token = toolbox.create_token(request)
    # Get the download url
    download_url = reverse('mydownloads:serve', kwargs={'token': token})
    # Send out an email containing the link
    notifications.send_file_ready(request.requester, download_url, request.filename)Add a management command to callprocess_file_requests, drop it in crontab and you’re good to go.But wait! There’s more!Using the FullPatternbgfilesincludes common patterns to structure your logic and minimize the code duplication. As you can see above
their usage is entirely optional.In this example we’ll use thebgfiles.patterns.FullPatternto render a template response when the file creation is
delayed and send out an email notification when the file is ready.Here’s our export handler class:class LoginEventExport(FullPattern):
    # These can all be overridden by get_* methods, e.g. get_file_type
    file_type = 'login_events'
    content_type = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
    delayed_template_name = 'reports/delayed_response.html'
    email_subject = _('File %(filename)s is ready!')
    email_text_message = _('Come and get it: %(download_url)s')

    def get_items(self, criteria):
        # Our default criteria class provides the request's QueryDict (e.g. request.POST) as `raw` and the
        # requester as `user`. This method is used for both online and offline selection of the items we want to
        # use.
        form = LoginExportFilterForm(data=criteria.raw)
        if not form.is_valid():
            # If the form is invalid we raise an exception so our view knows about it and can show the errors.
            # If the form became invalid while we're offline... well, that shouldn't happen.
            raise InvalidForm(form=form)
        # Valid form: apply our filters and return our events
        return form.apply_filters(LoginEvent.objects.all())

    def evaluate_dataset(self, dataset, http_request):
        # When we've got our dataset, including our criteria and items, we need to evaluate whether we can
        # deliver it right now or need to delay it. Let's use our magic number.
        # Note that this is only called during an HTTP request.
        dataset.delay = dataset.items.count() > 10000

    def write_bytes(self, dataset, buffer):
        # This is where we write our dataset to the buffer. What goes on in here depends on your dataset, type of
        # file and so on.Now let’s adapt our view to use it:def export_login_events(request):
    if request.method == 'POST':
        try:
            delayed, response = LoginEventExport('login-events.xlsx').respond_to(request)
            return response
        except InvalidForm as exc:
            form = exc.form
    else:
        form = LoginExportFilterForm()
    return render(request, 'reports/login_events_filter.html', context={'form': form})Here’s what happening:We let our export class handle the response when it’s a POST requestIt builds our dataset by wrapping the criteria (so we can use the same thing for both online and offline file generation) and fetching the items usingget_itemsbased on those criteriaIt then lets you evaluate the dataset to decide on what to do nextIf we don’t delay the file creation, the pattern will write our bytes to aHttpResponsewith the specified filename and content type.But when wedodelay the creation, it will:Add abgfiles.models.FileRequestto the databaseAsk you to schedule the request using itsschedulemethodRespond with a template response using thedelayed_template_nameTheschedulemethod does nothing by default, but if you use the included management command you can still have a cron
job process the outstanding requests automatically. If you prefer to use Celery, you can use thebgfiles.patterns.celery.ScheduleWithCeleryPatternclass instead. It subclasses theFullPatternclass.This has our online part covered, but we still need to adapt our cron job. Here’s what’s left of it using the pattern:def process_login_events(request):
    LoginEventExport(request.filename).create_file(request)That’s it. The default implementation ofcreate_filewill:Restore our criteria using thecriteria_classspecified on our exporterCallget_itemsusing those criteriaCallwrite_bytesto generate the file contentsHook up the contents to ourFileRequestand mark it as finishedSend out an email notification to the requesterBut we can still improve. Read on!Using the management commandThe included management commandbgfilesallows you to clean up expired file requests, whether you use the included patterns or not:$ python manage.py bgfiles clean --timeout=60 --sleep=1The above command will clean expired file requests, but will stop after a minute (or close enough) and go to sleep
for a second in between requests. By default it will also ignore file requests that have expired less than an
hour ago as to not interrupt any ongoing last-minute downloads. You can override this using the--leewayparameter.You can also use the management command to process outstanding requests. To do this you’ll need to register your
exporter in the registry:from bgfiles.patterns import FullPattern, registry

class LoginEventExport(FullPattern):
    # Same as above


# Register our class to process requests for our file type.
# You might want to place this in your AppConfig.ready.
registry.register(LoginEventExport, [LoginEventExport.file_type])Now all that’s needed to process file requests is to call the management command:$ python manage.py bgfiles process --sleep=1 --timeout=600 --items=20This will process our outstanding requests by looking up the correct handler in the registry and calling the handler’shandleclassmethod which by default will restore a class instance (using the pattern’srestoreclassmethod) and call itscreate_filemethod.Serving the fileWe’ve included a view you can use to serve the file. It will verify the token is valid, not expired and, by default,
check that the accessing user is also the user that requested the file. It doesn’t provide any decent error messages
in case something is wrong, so you might want to wrap it with your own view:from bgfiles.views import serve_file, SuspiciousToken, SignatureHasExpired, UserIsNotRequester

def serve(request, token):
    try:
        return serve_file(request, token, require_requester=True, verify_requester=True)
    except SuspiciousToken:
        # The token is invalid.
        # You could reraise this so Django can warn you about this suspicious operation
        return render(request, 'errors/naughty.html')
    except SignatureHasExpired:
        # Actually a subclass of PermissionDenied.
        # But you might want to inform the user they're a bit late to the party.
        return render(request, 'errors/signature_expired.html')
    except UserIsNotRequester:
        # Also a PermissionDenied subclass.
        # So the user's email was intercepted or they forwarded the mail to someone else.
        # Set verify_requester to False to disable this behavior.
        return render(request, 'errors/access_denied.html')Allowing anyone to access a fileThat is anyone with a valid token. By defaultbgfileswill assume you only want to serve a file to the user that
requested it. If you want to serve the file to anyone with a valid token it’s just as easy.ManuallyIn our manual example we usedtoolbox.create_tokento create our token. This embeds the id of the requester in the
token. To create a token anyone can use, usetoolbox.create_general_tokeninstead. Of course, the other token can
also be used by anyone because the verification is done in the view:# When we only want to serve the file to the requester, we use this:
serve_file(request, token, require_requester=True, verify_requester=True)

# When we want to serve the file to anyone, we use this:
serve_file(request, token, require_requester=False, verify_requester=False)Using patternsYou can tell the pattern to use “general” tokens by setting therequester_onlyclass variable toFalseor
by letting theis_requester_onlymethod returnFalse. The changes to the view are the same as above.Changing the signerBy defaultbgfilesuses Django’s signing module to handle tokens. Configuring the signer can be done by changing settings:BGFILES_DEFAULT_KEY: defaults toSECRET_KEYBGFILES_DEFAULT_SALT: defaults tobgfiles.We recommend specifying your own salt.BGFILES_DEFAULT_SERIALIZER: defaults to theJSONSerializerclass included in the toolboxBGFILES_DEFAULT_MAX_AGE: defaults to 86400 seconds (or a day)If you need a custom defaultSigner, you can setBGFILES_DEFAULT_SIGNERto an instance of your signer class.Of course, you should think about when to use the default signing method and settings and when to deviate and use a
custom one. Just be sure to use the same signer configuration when creating a token and when accepting a token:token = toolbox.create_token(request, signer=my_signer)

# And in your view
def serve(request, token):
    try:
        return serve_file(request, token, require_requester=True, verify_requester=True, signer=my_signer)
    except Stuff:
        # Error handling goes hereSpecifying a custom signer on a pattern class:class MyExporter(FullPattern):
    signer = my_signerOr:class MyExporter(FullPattern):

    def get_signer(self):
        return my_signerHistoryHistory is not collected yet. This is alpha software.