Bug: Sync has been failing for 3 days - recurring periodic problem - "Origin is not allowed to make this request"
Status: RESOLVED
Priority: --
Resolution: FIXED
Created: 2022-12-16 02:36:25+00:00
Resolved: 2023-01-02 22:43:54+00:00

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0

Steps to reproduce:

Nothing other than stayed with default sync settings.


Actual results:

I received notifications that sync was failing.  I turned on debugging in the logs.  The sync has been failing continuously for 3 days now.  I have noticed this in the past and it will fail for 7 or more days at a time and then resolve itself for a while.  This is not good behavior because one cannot rely on sync.

In the logs, there is a message "Origin is not allowed to make this request."  So that might be the issue.

1671152244260	Sync.ErrorHandler	DEBUG	tabs failed: {"data":"Origin is not allowed to make this request.","url":"https://sync-1-us-west1-g.sync.services.mozilla.com/1.5/166035168/storage/tabs?batch=true&commit=true","status":400,"success":false,"headers":{"alt-svc":"clear","content-length":"43","date":"Fri, 16 Dec 2022 00:57:24 GMT","server":"openresty/1.15.8.2","via":"1.1 google","x-firefox-spdy":"h2"},"failureCode":"error.engine.reason.record_upload_fail"}

I also tried connecting to the server directly with curl and I got a MAX_CONCURRENT_STREAMS == 100 message, so maybe this is a load balancer issue?

curl -vvv 'https://sync-1-us-west1-g.sync.services.mozilla.com/1.5/166035168/info/collections'
*   Trying 35.186.227.140...
* TCP_NODELAY set
* Connected to sync-1-us-west1-g.sync.services.mozilla.com (35.186.227.140) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=sync-1-us-west1-g.sync.services.mozilla.com
*  start date: Nov 20 02:15:43 2022 GMT
*  expire date: Feb 18 02:15:42 2023 GMT
*  subjectAltName: host "sync-1-us-west1-g.sync.services.mozilla.com" matched cert's "sync-1-us-west1-g.sync.services.mozilla.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbfff00d600)
> GET /1.5/166035168/info/collections HTTP/2
> Host: sync-1-us-west1-g.sync.services.mozilla.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 401 
< server: openresty/1.15.8.2
< date: Fri, 16 Dec 2022 00:46:28 GMT
< content-type: application/json
< content-length: 1
< x-weave-timestamp: 1671151588.13
< via: 1.1 google
< alt-svc: clear
< 
* Connection #0 to host sync-1-us-west1-g.sync.services.mozilla.com left intact
0* Closing connection 0



Expected results:

Sync should have succeeded and it should have written success-sync-* instead of error-sync-*