Repository: getsentry/snuba

Issue:
Segmentation Fault when running bootstrap
So I've seen this happen in a couple different scenarios. I'm trying to run Sentry in Kubernetes. After getting everything installed, I go to bootstrap Snuba (create the Kafka topics). I've experienced this with both Confluent Kafka as well as Apache Kafka (and multiple versions of each). I've also experienced this in both Minikube and AWS EKS clusters. 

```
/usr/src/snuba# LOG_LEVEL=debug snuba bootstrap --force
2020-01-08 18:39:41,151 Using Kafka with ('kafka-cp-kafka-headless.kafka:9092',)
2020-01-08 18:39:41,165 Attempting to connect to Kafka (attempt 0)
Segmentation fault (core dumped)
```


But if I add some debug log statements, it starts to work....
Here is my `git diff` which caused it to work suddenly 

```
diff --git a/snuba/cli/bootstrap.py b/snuba/cli/bootstrap.py
index 28f52f8..23a85fb 100644
--- a/snuba/cli/bootstrap.py
+++ b/snuba/cli/bootstrap.py
@@ -35,7 +35,6 @@ def bootstrap(
     if kafka:
         logger.debug("Using Kafka with %r", bootstrap_server)
         from confluent_kafka.admin import AdminClient, NewTopic

         attempts = 0
         while True:
             try:
@@ -58,6 +57,7 @@ def bootstrap(
                 time.sleep(1)

         topics = []
+        logger.debug("Made Connection Successfully")
         for name in DATASET_NAMES:
             dataset = get_dataset(name)
             table_writer = dataset.get_table_writer()
@@ -71,14 +71,14 @@ def bootstrap(
                             replication_factor=topic_spec.replication_factor,
                         )
                     )
-
+        print("Created Topics")
         for topic, future in client.create_topics(topics).items():
             try:
                 future.result()
                 logger.info("Topic %s created", topic)
             except Exception as e:
                 logger.error("Failed to create topic %s", topic, exc_info=e)
-
+        print("Actually created topics now")
     from snuba.clickhouse.native import ClickhousePool

     attempts = 0
```

It started to work after the 3rd log statement was added. 

Has anyone else experienced this?


Discussion:
Hi @nickrichardson-presto, I'm running into the same situation right now... Have you found another valid workaround, e.g. via CLI? I would like to stay on the original snuba image.
I found that editing that file above and adding the print statements fixes it every time. No idea why
I see the same behaviour indeed ! Really weird. Has any of you guys made a public docker image ?

Thanks!
I have not. I’m running in kubernetes and just exec-ed into the pod, make the change, than ran bootstrap
Fair enough !
I pushed a temporary image on docker hub for those interested ;) 

https://hub.docker.com/layers/theomathieu/snuba/r1/images/sha256-61610b24aa5e6b8f04ea98f1eddc9027e5a751f0ca0d3cfb537759dc61e2b6f4
Ok, I've made more tests and I don't think those logs are changing anything.

I managed to complete the bootstrap process by force bruting `LOG_LEVEL=debug snuba bootstrap --force`.
So the image won't change anything.

I created a job that run the bootstrap command 40 times.`LOG_LEVEL=debug snuba bootstrap --force`. And it works 👍 


I found out that if you execute the command with parameter `--no-kafka` at first something happens. Unfortunately I haven't figured out if that will lead to any further restrictions or issues with sentry... At least the `sentry_local` table is created this way.

`snuba bootstrap --no-kafka --force --log-level debug || snuba bootstrap --force --log-level debug || snuba migrate --log-level debug || echo true`
>I found out that if you execute the command with parameter --no-kafka at first something happens. Unfortunately I haven't figured out if that will lead to any further restrictions or issues with sentry... At least the sentry_local table is created this way.

I think there's an issue with waiting on Kafka topic creation futures and I'm working on a patch to hopefully resolve this. Stay tuned.

Resolving patch:
diff --git a/snuba/cli/bootstrap.py b/snuba/cli/bootstrap.py
--- a/snuba/cli/bootstrap.py
+++ b/snuba/cli/bootstrap.py
@@ -57,22 +57,28 @@ def bootstrap(
                     raise
                 time.sleep(1)
 
-        topics = []
+        topics = {}
         for name in DATASET_NAMES:
             dataset = get_dataset(name)
             table_writer = dataset.get_table_writer()
             if table_writer:
                 stream_loader = table_writer.get_stream_loader()
                 for topic_spec in stream_loader.get_all_topic_specs():
-                    topics.append(
-                        NewTopic(
-                            topic_spec.topic_name,
-                            num_partitions=topic_spec.partitions_number,
-                            replication_factor=topic_spec.replication_factor,
-                        )
+                    if topic_spec.topic_name in topics:
+                        continue
+                    logger.debug(
+                        "Adding topic %s to creation list", topic_spec.topic_name
+                    )
+                    topics[topic_spec.topic_name] = NewTopic(
+                        topic_spec.topic_name,
+                        num_partitions=topic_spec.partitions_number,
+                        replication_factor=topic_spec.replication_factor,
                     )
 
-        for topic, future in client.create_topics(topics).items():
+        logger.debug("Initiating topic creation")
+        for topic, future in client.create_topics(
+            list(topics.values()), operation_timeout=1
+        ).items():
             try:
                 future.result()
                 logger.info("Topic %s created", topic)
diff --git a/snuba/cli/migrate.py b/snuba/cli/migrate.py
--- a/snuba/cli/migrate.py
+++ b/snuba/cli/migrate.py
@@ -23,13 +23,15 @@ def migrate(*, log_level: str, dataset_name: Optional[str]) -> None:
     logging.basicConfig(
         level=getattr(logging, log_level.upper()), format="%(asctime)s %(message)s"
     )
+
+    if not local_dataset_mode():
+        logger.error("The migration tool can only work on local dataset mode.")
+        sys.exit(1)
+
     dataset_names = [dataset_name] if dataset_name else DATASET_NAMES
     for name in dataset_names:
         dataset = get_dataset(name)
         logger.info("Migrating dataset %s", name)
-        if not local_dataset_mode():
-            logger.error("The migration tool can only work on local dataset mode.")
-            sys.exit(1)
 
         clickhouse = Client(
             host=settings.CLICKHOUSE_HOST, port=settings.CLICKHOUSE_PORT,


Test patch: