Repository: googleapis/google-cloud-python

Issue:
Facing memory issue while loading the bigquery table data through python.
Hi, while trying to fetch the data of 1.8 M rows from Bigquery to local machine, we are getting Memory Error. 

Import pandas_gbq as pgbq
 df= pgbq.read_gbq(sql, dialect) 


Discussion:
I'm sorry you are having this issue.  Unfortunately, `pandas_gbq` isn't maintained in this repository:  you probably need to report the issue [over there](https://github.com/pydata/pandas-gbq/issues/new).

As a check, can you try iterating over the results the same query using just `google-cloud-bigquery` facilities?  E.g.:

```python
from google.cloud.bigquery import Client

client = Client()
query = client.query(sql)
row_count = 0 
for row in query:
    row_count += 1
```

You could also try converting to a dataframe directly using `google-cloud-bigquery`:

```python

from google.cloud.bigquery import Client

client = Client()
query = client.query(sql)
dataframe = query.to_dataframe()
```
@tseaver : Hi Tres, I have tried to implement your suggestion but now it is showing 
IOError: [Errno 22] Invalid argument

@HarshCHhz Are you using the `legacy` dialect for your query?  Also, can you please provide the full traceback?
@tseaver : I am using standard dialect for the query.
Traceback (most recent call last):
  File "filename_path.py", line 416, in <module>
    args.specialstrings
  File "filename_path.py", line 362, in main
    run()
  File "filename_path.py", line 288, in run
    sourcewisefilename= generatecsv(sqlinput[i], csvfilename,title[i])
  File "filename_path.py", line 82, in generatecsv
    client = Client()
  File "C:\Python27\lib\site-packages\google\cloud\bigquery\client.py", line 127, in __init__
    project=project, credentials=credentials, _http=_http)
  File "C:\Python27\lib\site-packages\google\cloud\client.py", line 216, in __init__
    Client.__init__(self, credentials=credentials, _http=_http)
  File "C:\Python27\lib\site-packages\google\cloud\client.py", line 129, in __init__
    credentials, _ = google.auth.default()
  File "C:\Python27\lib\site-packages\google\auth\_default.py", line 277, in default
    credentials, project_id = checker()
  File "C:\Python27\lib\site-packages\google\auth\_default.py", line 138, in _get_explicit_environ_credentials
    os.environ[environment_vars.CREDENTIALS])
  File "C:\Python27\lib\site-packages\google\auth\_default.py", line 63, in _load_credentials_from_file
    with io.open(filename, 'r') as file_obj:
IOError: [Errno 22] Invalid argument: '=path.json'

@HarshCHhz You need to set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to point to the full path to your credentials file.
I have already provided it in the path
.

@HarshCHhz according to the traceback, the environment variable contains `=path.json`.  Maybe you used `==` when setting it?

```bash
$ export GOOGLE_APPLICATION_CREDENTIALS=path.json
```
@tseaver I have made changes but still after it, again I am facing memory error issue over here.
@HarshCHhz do you see the MemoryError when iterating row-by-row, or only when converting the query to a dataframe?
After converting to dataframe

On Thu, Aug 30, 2018, 7:31 PM Tres Seaver <notifications@github.com> wrote:

> @HarshCHhz <https://github.com/HarshCHhz> do you see the MemoryError when
> iterating row-by-row, or only when converting the query to a dataframe?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5859#issuecomment-417329968>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AfhePWk3Njukbd1fKDobzwR3nw2wwjaJks5uV_A2gaJpZM4WRLLj>
> .
>

OK, looking at `google.cloud.bigquery.table.RowIterator.to_dataframe`, I can see that it is pulling the whole rowset into memory before constructing the `pandas.Dataframe`.  @HarshCHhz can you try with the following patch applied?

```patch
--- a/bigquery/google/cloud/bigquery/table.py
+++ b/bigquery/google/cloud/bigquery/table.py
@@ -1166,9 +1166,9 @@ class RowIterator(HTTPIterator):
             raise ValueError(_NO_PANDAS_ERROR)
 
         column_headers = [field.name for field in self.schema]
-        rows = [row.values() for row in iter(self)]
+        rows = (row.values() for row in iter(self))
 
-        return pandas.DataFrame(rows, columns=column_headers)
+        return pandas.DataFrame.from_records(rows, columns=column_headers)
 
 
 class _EmptyRowIterator(object):
```

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/table.py b/bigquery/google/cloud/bigquery/table.py
--- a/bigquery/google/cloud/bigquery/table.py
+++ b/bigquery/google/cloud/bigquery/table.py
@@ -1166,7 +1166,8 @@ def to_dataframe(self):
             raise ValueError(_NO_PANDAS_ERROR)
 
         column_headers = [field.name for field in self.schema]
-        rows = [row.values() for row in iter(self)]
+        # Use generator, rather than pulling the whole rowset into memory.
+        rows = (row.values() for row in iter(self))
 
         return pandas.DataFrame(rows, columns=column_headers)
 


Test patch: