Repository: LMFDB/lmfdb

Issue:
CMF download fails for newform with very high dimension
The download all data link for 983.2.c.a throws a server error (the other download links seem to work)

https://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/download_newform/983.2.c.a


Discussion:
What should we do about it?
We shouldn't offer download links that we know are not going to work, so if the issue is simply that there is too much data (as opposed to an actual bug) we should not offer that link.  I notice that the download all embeddings and Satake angle links do not throw server errors, but they also don't work in that they eventually hit a network failure (after about 100 MB).

I don't think we should be offering the option to download all the embeddings (or Satake angles) in one link, at least not when the dimension is large.  The user can always pick their favorite embedding and download the coefficient data from the embedded CMF page (but I notice there is not a Satake angle download on those pages, is that intentional?)

I think the download all stored data link on the Galois orbit pages should not include embedding data (including Satake angles), or at least not when the dimension is large.
Here is another example of a download that is failing where the dimension is smaller ("only" 1360)

https://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/83/6/c/a/

and another with dimension 648

https://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/download_newform/673.2.m.a
Even after #4218 (which is on web), we seem to still be getting timeouts, e.g. on

https://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/download_newform/101.5.i.a

I don't understand why, because the amount of data is quite small.  The download works on beta (although it took a suspiciously long time the first time I tried it).
I don't understand it also.  Maybe once in a while goes for a sequential search? When I run the `explain analyze` on proddb and on grace I get different plans to do the search, on grace it decides to do a sequential search. But also doing a sequential search on grace is much faster but in either case no reason for a timeout.

This is on proddb
```
lmfdb=# EXPLAIN ANALYZE SELECT "an", "hecke_ring_numerators", "hecke_ring_denominators", "hecke_ring_inverse_numerators", "hecke_ring_inverse_denominators", "hecke_ring_cyclotomic_generator", "hecke_ring_character_values", "hecke_ring_power_basis", "maxp" FROM "mf_hecke_nf" WHERE "hecke_orbit_code" = 549839700069 LIMIT 1;
                                                                       QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..8.02 rows=1 width=1240) (actual time=0.173..0.174 rows=0 loops=1)
   ->  Index Scan using mf_hecke_nf_hecke_orbit_code_hash on mf_hecke_nf  (cost=0.00..8.02 rows=1 width=1240) (actual time=0.172..0.172 rows=0 loops=1)
         Index Cond: (hecke_orbit_code = '549839700069'::bigint)
         Rows Removed by Index Recheck: 1
 Planning time: 1.462 ms
 Execution time: 0.229 ms
(6 rows)
```
and this is on grace
```
lmfdb=# EXPLAIN ANALYZE SELECT "an", "hecke_ring_numerators", "hecke_ring_denominators", "hecke_ring_inverse_numerators", "hecke_ring_inverse_denominators", "hecke_ring_cyclotomic_generator", "hecke_ring_character_values", "hecke_ring_power_basis", "maxp" FROM "mf_hecke_nf" WHERE "hecke_orbit_code" = 549839700069 LIMIT 1;
                                                            QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=1000.00..36986.75 rows=1 width=1229) (actual time=75.512..103.098 rows=0 loops=1)
   ->  Gather  (cost=1000.00..36986.75 rows=1 width=1229) (actual time=75.510..103.095 rows=0 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Parallel Seq Scan on mf_hecke_nf  (cost=0.00..35986.65 rows=1 width=1229) (actual time=70.027..70.027 rows=0 loops=3)
               Filter: (hecke_orbit_code = '549839700069'::bigint)
               Rows Removed by Filter: 69237
 Planning time: 1.554 ms
 Execution time: 103.208 ms
(9 rows)
```

Most searches on www.lmfdb.org trigger an indexed search https://grace.mit.edu/munin/PostgreSQL/proddb/postgres_scans_lmfdb.html
Regardless, there are a couple of queries that go over 30s 
https://grace.mit.edu/munin/PostgreSQL/proddb/postgres_querylength_lmfdb.html
We could try to increase the memory on proddb to see if it helps with some of the caching.

We still seem to be getting problems here.  On the page https://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/389/2/d/a/ the download all data to text link times out consistently (but is nearly instantaneous on beta).  What confuses me about this is that I don't think the download should be doing any queries that aren't being done when the page is displayed, but we are never getting timeouts on the display of the page.

@edgarcosta I'm also confused by the timings you quoted above, since there should be zero matches for the hecke_orbit_code for the newforms where we are having problems.  But in any case, I don't see why the download should be doing queries that are any different from the queries that are done when the page is rendered, yet on production the download query is consistently timing out but the page render never does.

Those are very good points!

 First, I have no clue why some of these queries are timing out, as you
noted they should be identical to the ones for the webpage. We should check
that!

Second,we have removed all the troublesome urls from the page for large
dimension, but the route is still valid, and the download code doesn't
check for dimension.
We should fix that

On Thu, Dec 10, 2020, 05:43 Andrew Sutherland <notifications@github.com>
wrote:

> We still seem to be getting problems here. On the page
> https://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/389/2/d/a/ the
> download all data to text link times out consistently (but is nearly
> instantaneous on beta). What confuses me about this is that I don't think
> the download should be doing any queries that aren't being done when the
> page is displayed, but we are never getting timeouts on the display of the
> page.
>
> @edgarcosta <https://github.com/edgarcosta> I'm also confused by the
> timings you quoted above, since there should be zero matches for the
> hecke_orbit_code for the newforms where we are having problems. But in any
> case, I don't see why the download should be doing queries that are any
> different from the queries that are done when the page is rendered, yet on
> production the download query is consistently timing out but the page
> render never does.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/LMFDB/lmfdb/issues/4205#issuecomment-742439574>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACO2BVCOJDRMZ66SMXY6BTSUCQ4ZANCNFSM4TFZ7CNQ>
> .
>


Resolving patch:
diff --git a/lmfdb/classical_modular_forms/download.py b/lmfdb/classical_modular_forms/download.py
--- a/lmfdb/classical_modular_forms/download.py
+++ b/lmfdb/classical_modular_forms/download.py
@@ -300,12 +300,9 @@ def download_newform(self, label, lang='text'):
         if data is None:
             return abort(404, "Label not found: %s"%label)
         form = WebNewform(data)
-        form.setup_cc_data({'m':'1-%s'%form.dim})
         if form.has_exact_qexp:
             data['qexp'] = form.qexp
             data['traces'] = form.texp
-        #if form.has_complex_qexp:
-        #    data['complex_embeddings'] = form.cc_data
         return self._wrap(Json.dumps(data),
                           label,
                           lang=lang,


Test patch: