Repository: dj-stripe/dj-stripe

Issue:
Error when creating SubscriptionItems for metered plans
Python==3.7.1
Django==2.1.7
djstripe==2.0.0

The issue seems to arise when processing the `customer.subscription.created` webhook callback. dj-stripe attempts to create the `SubscriptionItem`s listed in the Stripe response. The Stripe response for `Subscription` items that are on metered plans does not contain the `quantity` attribute, thus when dj-stripe ingests that data it fails to create a `SubscriptionItem` as the `quantity` field does not accept null values nor does it have a default. Here is a snippet copied from my Stripe logs of what was POSTed to my webhook endpoint:
```
"items": {
      "object": "list",
      "data": [
        {
          "id": "si_*************",
          "object": "subscription_item",
          "billing_thresholds": null,
          "created": 1552418209,
          "metadata": {
          },
          "plan": {
            "id": "plan_*************",
            "object": "plan",
            "active": true,
            "aggregate_usage": "sum",
            "amount": 100,
            "billing_scheme": "per_unit",
            "created": 1552375552,
            "currency": "usd",
            "interval": "day",
            "interval_count": 1,
            "livemode": false,
            "metadata": {
            },
            "nickname": "Per completing record",
            "product": "prod_*************",
            "tiers": null,
            "tiers_mode": null,
            "transform_usage": null,
            "trial_period_days": null,
            "usage_type": "metered"
          },
          "subscription": "sub_*************"
        },
        {
          "id": "si_*************",
          "object": "subscription_item",
          "billing_thresholds": null,
          "created": 1552418209,
          "metadata": {
          },
          "plan": {
            "id": "plan_*************",
            "object": "plan",
            "active": true,
            "aggregate_usage": null,
            "amount": 200,
            "billing_scheme": "per_unit",
            "created": 1552369666,
            "currency": "usd",
            "interval": "day",
            "interval_count": 1,
            "livemode": false,
            "metadata": {
            },
            "nickname": "Per worker",
            "product": "prod_*************",
            "tiers": null,
            "tiers_mode": null,
            "transform_usage": null,
            "trial_period_days": null,
            "usage_type": "licensed"
          },
          "quantity": 0,
          "subscription": "sub_*************"
        }
      ],
      "has_more": false,
      "total_count": 2,
      "url": "/v1/subscription_items?subscription=sub_*************"
    },
```
You'll see that for the first (metered plan) `SubscriptionItem` the `quantity` attribute is missing. I believe this is causing the error I'm seeing:
```
psycopg2.IntegrityError: null value in column "quantity" violates not-null constraint
```

Not entirely sure what the remedy is here, [Stripe documentation for SubscriptionItem](https://stripe.com/docs/api/subscription_items/object#subscription_item_object-quantity) suggests that this attribute should be a positive integer or 0, but doesn't specify that the attribute is always present. dj-stripe may need to be modified to handle this case?

Everything works fine if I do not include the metered plan in the subscription.

Has anyone else dealt with this issue before?


Discussion:
Note: the code below uses the Stripe API alongside dj-stripe, but there is likely a way to do this using just the dj-stripe code/objects as of v2. @therefromhere may be able to tell us, I don't know off the top off my head. 

Also, I'm just plucking off the first plan from the set Stripe Plans in the example below, be sure to set the subscription to the plan the user actually selected in the subscription form on your end before attempting to use this code. I'm also setting the trial_end parameter, in the event you've a trialing period and just haven't mentioned it yet - this is how you've ensure the trial begins - remove trial_end if the subscription is meant to be paid for, and thus the charge created, immediately upon successful instantiation of the subscription itself. 

```
token = request.POST['stripe_token']
stripe_plans = stripe.Plan.list()
trial_start = datetime.datetime.fromtimestamp(int(time.time())) 
trial_end = int(trial_start.timestamp()) + 2629743 #datetime.timedelta(days=30)

customer = stripe.Customer.create(
    email='john@doe.com',
    source=token
)

subscription = stripe.Subscription.create(
    customer=customer.id,
    items=[{
        'plan': stripe_plans[0]['id'], 
        'quantity': 1
    }],
    trial_end=int(trial_end)
)

# may be unnecessary or redundant as of dj-stripe v2, waiting on @therefromhere to advise/confirm
djstripe_customer = Customer.sync_from_stripe_data(customer)
djstripe_customer.subscriber = user
djstripe_customer.save()
```

So the question becomes: You're setting a quantity when creating the subscription, correct?
Yes, I am setting the quantity for the first subscription item to 0 when I create the subscription. It should be noted that I created the subscription from the Stripe dashboard. So this error is purely happening during the webhooks event callback. 
I've not looked into this deeply but it sounds like this could be a similar case to #773 - fudge the data on the stable 2.0 branch (which we won't allow migrations) using `_manipulate_stripe_object_hook` and add a migration on master that makes `SubscriptionItem.quantity nullable` (which would be part of a future 2.1 release).

I've a feeling there's a different PR which was more similar but I can't find it right now.
@jameschristopher - check your setup. I'm not sure why you want to instantiate a tiered subscription with a quantity of 0, but for gits and shiggles I went in and subscribed a customer to a tiered subscription with quantity=0, and the events/webhooks and updates to the application all went through and look fine on my end. 

Stripe API Version: 2018-11-08
[customer.subscription.created webhook](https://gist.github.com/blackboxoperations/537cb5e89052658c52ae9d3041556c16)
[customer.subscription.created event object contained in the webhook
](https://gist.github.com/blackboxoperations/e80c35792b74a42fd15621642e8dce4a)
Webhook is valid, processed. Event logged. If there's something wrong here, I can't recreate it on my end. Going to look at #773 and see if that sheds some light on this matter. 

Well the issue isn’t with tiered plans it’s with metered plans. 
@therefromhere I’ll give that a shot and update this thread
Cool.  Note that master and stable/2.0 are currently in sync since we didn't add any post-2.0 release breaking changes yet (migrations etc), so this would be a point where they'll diverge.
It seems that `_manipulate_stripe_object_hook` is not getting called during this issue. I was able to resolve the issue by fudging the `_stripe_object_to_subscription_items` method using `.setdefault` right before it calls the `_get_or_create_from_stripe_object` class method.
```
@classmethod
def _stripe_object_to_subscription_items(cls, target_cls, data, subscription):
	"""
	Retrieves SubscriptionItems for a subscription.

	If the subscription item doesn't exist already then it is created.

	:param target_cls: The target class to instantiate per invoice item.
	:type target_cls: ``SubscriptionItem``
	:param data: The data dictionary received from the Stripe API.
	:type data: dict
	:param invoice: The invoice object that should hold the invoice items.
	:type invoice: ``djstripe.models.Subscription``
	"""

	items = data.get("items")
	if not items:
	    return []

	subscriptionitems = []
	for item_data in items.get("data", []):
	    item_data.setdefault('quantity', 0)
	    item, _ = target_cls._get_or_create_from_stripe_object(item_data, refetch=False)
	    subscriptionitems.append(item)

	return subscriptionitems
```

 I also added a similar line to `_manipulate_stripe_object_hook` as suggested.

Resolving patch:
diff --git a/djstripe/models/billing.py b/djstripe/models/billing.py
--- a/djstripe/models/billing.py
+++ b/djstripe/models/billing.py
@@ -1199,6 +1199,7 @@ class SubscriptionItem(StripeModel):
 		related_name="subscription_items",
 		help_text="The plan the customer is subscribed to.",
 	)
+	# TODO - quantity should be nullable, since it's not set for metered plans
 	quantity = models.PositiveIntegerField(
 		help_text=("The quantity of the plan to which the customer should be subscribed.")
 	)
@@ -1209,6 +1210,14 @@ class SubscriptionItem(StripeModel):
 		help_text="The subscription this subscription item belongs to.",
 	)
 
+	@classmethod
+	def _manipulate_stripe_object_hook(cls, data):
+		# for metered plans quantity isn't set, so should be nullable
+		# arguably this should default to 1 instead of 0 (to match subscription)
+		data["quantity"] = data.get("quantity") or 0
+
+		return data
+
 
 class UsageRecord(StripeModel):
 	"""


Test patch:
diff --git a/tests/__init__.py b/tests/__init__.py
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -770,6 +770,27 @@ def capture(self):
 	],
 }
 
+FAKE_PLAN_METERED = {
+	"id": "plan_fakemetered",
+	"object": "plan",
+	"active": True,
+	"aggregate_usage": "sum",
+	"amount": 200,
+	"billing_scheme": "per_unit",
+	"created": 1552632817,
+	"currency": "usd",
+	"interval": "month",
+	"interval_count": 1,
+	"livemode": False,
+	"metadata": {},
+	"nickname": "Sum Metered Plan",
+	"tiers": None,
+	"tiers_mode": None,
+	"transform_usage": None,
+	"trial_period_days": None,
+	"usage_type": "metered",
+}
+
 
 class SubscriptionDict(dict):
 	def __setattr__(self, name, value):
@@ -778,7 +799,7 @@ def __setattr__(self, name, value):
 
 		# Special case for plan
 		if name == "plan":
-			for plan in [FAKE_PLAN, FAKE_PLAN_II]:
+			for plan in [FAKE_PLAN, FAKE_PLAN_II, FAKE_TIER_PLAN, FAKE_PLAN_METERED]:
 				if value == plan["id"]:
 					value = plan
 
@@ -925,6 +946,43 @@ def save(self, idempotency_key=None):
 )
 
 
+FAKE_SUBSCRIPTION_METERED = SubscriptionDict(
+	{
+		"id": "sub_1rn1dp7WgjMtx9",
+		"object": "subscription",
+		"application_fee_percent": None,
+		"billing": "charge_automatically",
+		"cancel_at_period_end": False,
+		"canceled_at": None,
+		"current_period_end": 1441907581,
+		"current_period_start": 1439229181,
+		"customer": "cus_6lsBvm5rJ0zyHc",
+		"discount": None,
+		"ended_at": None,
+		"metadata": {},
+		"items": {
+			"data": [
+				{
+					"created": 1441907581,
+					"id": "si_UXYmKmJp6aWTw6",
+					"metadata": {},
+					"object": "subscription_item",
+					"plan": deepcopy(FAKE_PLAN_METERED),
+					"subscription": "sub_1rn1dp7WgjMtx9",
+				}
+			]
+		},
+		"plan": deepcopy(FAKE_PLAN_METERED),
+		"quantity": 1,
+		"start": 1439229181,
+		"status": "active",
+		"tax_percent": None,
+		"trial_end": None,
+		"trial_start": None,
+	}
+)
+
+
 class Sources(object):
 	def __init__(self, card_fakes):
 		self.card_fakes = card_fakes
diff --git a/tests/test_plan.py b/tests/test_plan.py
--- a/tests/test_plan.py
+++ b/tests/test_plan.py
@@ -9,10 +9,13 @@
 from django.test import TestCase
 
 from djstripe.admin import PlanAdmin
+from djstripe.enums import PlanUsageType
 from djstripe.models import Plan
 from djstripe.settings import STRIPE_SECRET_KEY
 
-from . import FAKE_PLAN, FAKE_PLAN_II, FAKE_TIER_PLAN, AssertStripeFksMixin
+from . import (
+	FAKE_PLAN, FAKE_PLAN_II, FAKE_PLAN_METERED, FAKE_TIER_PLAN, AssertStripeFksMixin
+)
 
 
 class TestPlanAdmin(TestCase):
@@ -116,6 +119,18 @@ def test_stripe_tier_plan(self, plan_retrieve_mock):
 			plan, expected_blank_fks={"djstripe.Customer.coupon", "djstripe.Plan.product"}
 		)
 
+	@patch("stripe.Plan.retrieve")
+	def test_stripe_metered_plan(self, plan_retrieve_mock):
+		plan_data = deepcopy(FAKE_PLAN_METERED)
+		plan = Plan.sync_from_stripe_data(plan_data)
+		self.assertEqual(plan.id, plan_data["id"])
+		self.assertEqual(plan.usage_type, PlanUsageType.metered)
+		self.assertIsNotNone(plan.amount)
+
+		self.assert_fks(
+			plan, expected_blank_fks={"djstripe.Customer.coupon", "djstripe.Plan.product"}
+		)
+
 
 class HumanReadablePlanTest(TestCase):
 	def test_human_readable_free_usd_daily(self):
diff --git a/tests/test_subscription.py b/tests/test_subscription.py
--- a/tests/test_subscription.py
+++ b/tests/test_subscription.py
@@ -14,8 +14,8 @@
 from djstripe.models import Plan, Subscription
 
 from . import (
-	FAKE_CUSTOMER, FAKE_CUSTOMER_II, FAKE_PLAN, FAKE_PLAN_II,
-	FAKE_SUBSCRIPTION, FAKE_SUBSCRIPTION_CANCELED,
+	FAKE_CUSTOMER, FAKE_CUSTOMER_II, FAKE_PLAN, FAKE_PLAN_II, FAKE_PLAN_METERED,
+	FAKE_SUBSCRIPTION, FAKE_SUBSCRIPTION_CANCELED, FAKE_SUBSCRIPTION_METERED,
 	FAKE_SUBSCRIPTION_MULTI_PLAN, AssertStripeFksMixin, datetime_to_unix
 )
 
@@ -464,3 +464,46 @@ def test_sync_multi_plan(
 
 		items = subscription.items.all()
 		self.assertEqual(2, len(items))
+
+		self.assert_fks(
+			subscription,
+			expected_blank_fks={
+				"djstripe.Customer.coupon",
+				"djstripe.Customer.subscriber",
+				"djstripe.Plan.product",
+				"djstripe.Subscription.plan",
+			},
+		)
+
+	@patch("stripe.Plan.retrieve")
+	@patch("stripe.Customer.retrieve", return_value=deepcopy(FAKE_CUSTOMER_II))
+	@patch(
+		"stripe.Subscription.retrieve", return_value=deepcopy(FAKE_SUBSCRIPTION_METERED)
+	)
+	def test_sync_metered_plan(
+		self, subscription_retrieve_mock, customer_retrieve_mock, plan_retrieve_mock
+	):
+		subscription_fake = deepcopy(FAKE_SUBSCRIPTION_METERED)
+		self.assertNotIn(
+			"quantity",
+			subscription_fake["items"]["data"],
+			"Expect Metered plan SubscriptionItem to have no quantity",
+		)
+
+		subscription = Subscription.sync_from_stripe_data(subscription_fake)
+
+		items = subscription.items.all()
+		self.assertEqual(1, len(items))
+
+		item = items[0]
+
+		self.assertEqual(subscription.quantity, 1)
+		# Note that subscription.quantity is 1, but item.quantity isn't set on metered plans,
+		# it probably should be nullable
+		self.assertEqual(item.quantity, 0)
+		self.assertEqual(item.plan.id, FAKE_PLAN_METERED["id"])
+
+		self.assert_fks(
+			subscription,
+			expected_blank_fks={"djstripe.Customer.coupon", "djstripe.Plan.product"},
+		)