Repository: dj-stripe/dj-stripe

Issue:
Sync all Stripe data to local
Should we have a management command that sync all Stripe data to local db?
Ability to manually pull events as an alternative to webhook
I'm at early stage of development and I didn't have any setup where I could use webhook to get the events from Stripe. What I did was to manually poll the events and then shove it to djstripe as if it's receiving a webhook event. I'm not sure if there's an even simpler way or if this has been implemented somewhere already. Do you think this is going to be useful?

Here's the code I'm using:

```
    @classmethod
    def handle_event(cls, data):
        if Event.objects.filter(stripe_id=data["id"]).exists():
            EventProcessingException.objects.create(
                data=data,
                message="Duplicate event record",
                traceback=""
            )
        else:
            event = Event.objects.create(
                stripe_id=data["id"],
                kind=data["type"],
                livemode=data["livemode"],
                webhook_message=data
            )
            event.validate()
            event.process()

    @classmethod
    def poll_events(cls):
        """
        Polls events from Stripe - useful when you can't use a webhook.
        """
        try:
            # Try to get the latest entry
            latest_id = Event.objects.order_by('-created', '-id')[0].stripe_id
        except IndexError:
            # We don't have the first entry, so go find it.
            events = stripe.Event.all(limit=1, include=['total_count'])
            total_count = events['total_count'] 
            if total_count == 0:
                return # no events yet
            else:                
                # Stripe, unfortunately, doesn't have an API to get the  first 
                # entry directly, so what we can do is to jump towards the end 
                # and walk the list from there. Using offset alone is not 
                # enough, since it's possible that more events got generated 
                # in the interim.
                events = stripe.Event.all(limit=1, offset=total_count-1)
                first = events['data'][-1]
                while (events['has_more']):
                    events = stripe.Event.all(limit=100, starting_after=first.id)
                    first = events['data'][-1]

                logger.info(
                    'First Polled Event: evt_id=%s evt_type=%s created=%s' % 
                    (first.id, first.type, first.created))            

                cls.handle_event(first.to_dict())
                latest_id = first.id

        # Process each event in chronological order
        events = stripe.Event.all(limit=10, ending_before=latest_id)
        for evt in reversed(events['data']):     

            logger.info('Polled Event: evt_id=%s evt_type=%s created=%s' % 
                    (evt.id, evt.type, evt.created))            

            cls.handle_event(evt.to_dict())
```



Discussion:
This is #89. Happy to take a PR for such a management command, yes.
It could be useful, but I'm not sure where this code would go.

I think it can go to the Event class. The first function is actually copied from the webhook view. If we move that code into the Event class, then and it would be shared by the two input sources. Let me know if this works and I'll include this in my pull request.

Okay, your suggestion is that the code gets attached to the Event model. Let me dwell on it for a bit, and I'll give an answer later today.

Hi mlvn23, sorry it took us so long to respond on this.

Seems like a nice, simple alternative. Please create a pull request with tests and documentation. The docs should probably go in the cookbook, but if you can think of a better place for it, let me know.

If everything's solid, I'll merge it.

Just an add-on thought here. The idea of polling for events and processing ones that have not already been processed does not only apply to people not using webhooks.

The idea could also be a failsafe against rare cases where webhook event processing fails (e.g. when there is a service outage at Stripe.com). In order to really make event processing as reliable as possible, it's probably a good idea to poll for events in addition to handling webhooks and check for events that are valid but not yet processed. Such events in the database would probably cases where it would be a good idea to retreive the Event data and attempt processing (again).

:+1: 

I'd also like to point out that while our current syncing system works, it seems incomplete and can sometimes be unreliable, especially if the webhook endpoint isn't up 100% of the time.

There should be a master sync util that gets called after each model "update" action in the djstripe code. I'm fairly sure we do so in almost all cases, but it lacks consistency. It should also keep track of the last sync datetime so that syncing doesn't take forever every time. Something like `djstripe_sync_models(models=["Customer", "Subscription", "Invoice" ... ])`

I'm using a modified version of this to pull information from Stripe.

I've made it a management command as I don't expect it will be used very often..

I've run into some issues with this: for instance, a "customer.source.updated" event that adds a card that has since been removed. In that case, retrieving the card via the API fails. I'm not sure how to handle that yet.. For now I'm just ignoring the errors, on the theory that I don't really care about the details of a card that's gone away.

UPDATE: although I'd love to see something like this work one day, I wouldn't recommend it at present. At minimum you'll have to deal with historical inconsistencies (eg, a "customer.source created" event for a source that has since been deleted - you can't check it against the API because it doesn't exist now); if you've got any djstripe_subscriber metadata, that can cause problems too.

```
import logging

import stripe
from djstripe.models import Event

from django.core.management.base import BaseCommand

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

class Command(BaseCommand):
    """Replay the Event stream"""

    help = """Replay the Event stream"""

    def _find_first_event(self):
        try:
            latest_id = Event.objects.order_by('-created', '-djstripe_id')[0].stripe_id
        except IndexError:
            latest_id = None

        if not latest_id:
            # We don't have the first entry, so go find it
            events = stripe.Event.all(limit=1, include=['total_count'])
            total_count = events['total_count'] 
            if total_count == 0:
                return latest_id # no events yet
            else:
                # Stripe, unfortunately, doesn't have an API to get the  first 
                # entry directly, so what we can do is to jump towards the end 
                # and walk the list from there. Using offset alone is not 
                # enough, since it's possible that more events got generated 
                # in the interim.
                events = stripe.Event.all(limit=1, offset=total_count-1)
                first = events['data'][-1]
                while (events['has_more']):
                    events = stripe.Event.all(limit=100, starting_after=first.id)
                    first = events['data'][-1]

                logger.info(
                    'First Polled Event: event_id=%s event_type=%s created=%s' % 
                    (first.id, first.type, first.created))

                self._handle_event(first)

            latest_id = first.id
            return latest_id

    def _handle_event(self, event):
        try:
            obj = Event().process(event)
            obj.save()
            logger.debug("Created event {}".format(obj.stripe_id))
        except stripe.error.InvalidRequestError:
            # Likely due to a customer.source which was created in this event
            # but has since been deleted, so can't now be retrieved
            logger.warn("Couldn't process event: {}\n".format(event.id))
        except django.models.DoesNotExist as e:
            ## an old event with djstripe_subscriber metadata that isn't relevant
            if e.msg == "Customer matching query does not exist":
                pass
            

    def _handle_events(self, events):
        for event in reversed(events['data']):
            logger.info('Polled Event: event_id=%s event_type=%s created=%s' %
                        (event.id, event.type, event.created))
            self._handle_event(event)

    def handle(self, *args, **options):
        """Replay the Event stream"""

        latest_id = self._find_first_event()

        while latest_id:
            events = stripe.Event.all(limit=100, ending_before=latest_id)
            self._handle_events(events)
            if events['has_more']:
                latest_id = events['data'][1]
            else:
                latest_id = None
                

```
    
I want to do a management command that pull
```
Customer
Subscribe
Card
Invoice
```
From Stripe to local. Can I make this and open a PR?
I'll try and do some form of this for 2.1.0, since it'd be useful for re-syncing Account with the changes from flat to the nested JSON fields - see https://stripe.com/docs/upgrades#2019-02-19 / #907

Resolving patch:
diff --git a/djstripe/management/commands/djstripe_sync_models.py b/djstripe/management/commands/djstripe_sync_models.py
new file mode 100644
--- /dev/null
+++ b/djstripe/management/commands/djstripe_sync_models.py
@@ -0,0 +1,89 @@
+from typing import List
+
+from django.apps import apps
+from django.core.management.base import BaseCommand, CommandError
+
+from ... import models, settings
+
+
+class Command(BaseCommand):
+	"""Sync models from stripe."""
+
+	help = "Sync models from stripe."
+
+	def add_arguments(self, parser):
+		parser.add_argument(
+			"args",
+			metavar="ModelName",
+			nargs="*",
+			help="restricts sync to these model names (default is to sync all supported models)",
+		)
+
+	def handle(self, *args, **options):
+		app_label = "djstripe"
+		app_config = apps.get_app_config(app_label)
+		model_list = []  # type: List[models.StripeModel]
+
+		if args:
+			for model_label in args:
+				try:
+					model = app_config.get_model(model_label)
+				except LookupError:
+					raise CommandError("Unknown model: {}.{}".format(app_label, model_label))
+
+				model_list.append(model)
+		else:
+			model_list = app_config.get_models()
+
+		for model in model_list:
+			self.sync_model(model)
+
+	def sync_model(self, model):
+		model_name = model.__name__
+
+		if not issubclass(model, models.StripeModel):
+			print("Skipping {} (not a StripeModel)".format(model_name))
+			return
+
+		if model.stripe_class is None:
+			print("Skipping {} (no stripe_class)".format(model_name))
+			return
+
+		if not hasattr(model.stripe_class, "list"):
+			print("Skipping {} (no stripe_class.list)".format(model_name))
+			return
+
+		print("Syncing {}:".format(model_name))
+
+		try:
+			count = 0
+
+			if model is models.Account:
+				# special case, since own account isn't returned by Account.api_list
+				stripe_obj = models.Account.stripe_class.retrieve(
+					api_key=settings.STRIPE_SECRET_KEY
+				)
+				count += 1
+				djstripe_obj = model.sync_from_stripe_data(stripe_obj)
+				print(
+					"  id={id}, pk={pk} ({djstripe_obj})".format(
+						id=djstripe_obj.id, pk=djstripe_obj.pk, djstripe_obj=djstripe_obj
+					)
+				)
+
+			for stripe_obj in model.api_list():
+				count += 1
+				djstripe_obj = model.sync_from_stripe_data(stripe_obj)
+				print(
+					"  id={id}, pk={pk} ({djstripe_obj})".format(
+						id=djstripe_obj.id, pk=djstripe_obj.pk, djstripe_obj=djstripe_obj
+					)
+				)
+
+			if count == 0:
+				print("  (no results)")
+			else:
+				print("  Synced {count} {model_name}".format(count=count, model_name=model_name))
+
+		except Exception as e:
+			print(e)


Test patch: