opened
cozy9
username_0
I have been running my script periodically, it throws the following error today, looks like the page returned from Google Scholar might have a different format now? Thanks!

Traceback (most recent call last):
  File "./crawl.py", line 27, in <module>
    for x in rslt:
  File "/usr/local/lib/python3.6/site-packages/scholarly.py", line 100, in _search_scholar_soup
    yield Publication(row, 'scholar')
  File "/usr/local/lib/python3.6/site-packages/scholarly.py", line 138, in __init__
    title = databox.find('h3', class_='gs_rt')
AttributeError: 'NoneType' object has no attribute 'find'
Did google scholar changes its format?
issue
created
cozy9
username_0
To reproduce it, use the following script:

```
import scholarly
results=scholarly.search_pubs_custom_url('/scholar?q=malaria+vaccine&hl=en')
for x in results:
    print(x.bib['title'][:50])
```

It outputs
```
Randomised trial of efficacy of SPf66 vaccine agai
Safety and immunogenicity in man of a synthetic pe
Efficacy of RTS, S/AS02 malaria vaccine against Pl
Genetically modified Plasmodium parasites as a pro
Traceback (most recent call last):
  File "./t.py", line 8, in <module>
    for x in results:
  File "/usr/local/lib/python3.6/site-packages/scholarly.py", line 100, in _search_scholar_soup
    yield Publication(row, 'scholar')
  File "/usr/local/lib/python3.6/site-packages/scholarly.py", line 138, in __init__
    title = databox.find('h3', class_='gs_rt')
AttributeError: 'NoneType' object has no attribute 'find'
```
comment
created
DownPeriscope
username_1
I'm also encountering the same issue on my end. Not sure yet as to what is causing it.
comment
created
cozy9
username_0
It seems Google Scholar page now contains a suggestion section at the end of the page, which produces an extra None Publication.  For now, my own patch is to edit the scholarly.py file in two places.

The bold face places are my edits
```
1.  Line 100
        for row in soup.find_all('div', 'gs_r'):
            pub=Publication(row, 'scholar')
            if len(pub.bib)==0: continue
            yield pub
2. Line 137
            databox = __data.find('div', class_='gs_ri')
            if (databox is None):
                return
            title = databox.find('h3', class_='gs_rt')
```

I am sure the author will be able to provide a cleaner fix.
comment
created
DownPeriscope
username_1
Thank you for the response. I implemented those changes but still no luck on my end unfortunately. I'm seeing this error when running the example from the author. This example here:

`# Retrieve the author's data, fill-in, and print
search_query = scholarly.search_author('Steven A Cholewiak')
author = next(search_query).fill()
print(author)

# Print the titles of the author's publications
print([pub.bib['title'] for pub in author.publications])

# Take a closer look at the first publication
pub = author.publications[0].fill()
print(pub)

# Which papers cited that publication?
print([citation.bib['title'] for citation in pub.get_citedby()])`

It's the final line where it is supposed to print titles of cited publications that seems to be giving the `NoneType` error.
comment
created
cozy9
username_0
Add one more then around 137:

```
            databox = __data.find('div', class_='gs_ri')
            if databox is None:
                self.bib={'title':None, 'url':None, 'author':None, 'abstract':None, 'eprint':None}
                return
            title = databox.find('h3', class_='gs_rt')
```

Again, my patch only prevents it from crashing, it returns publications entry with None title.  I don't know enough about Google Scholar to provide a patch that can fix the root cause.
comment
created
namp
username_2
I inspected the scholar pages and the following two changes did the trick for me:

1```
.  Line 100
        for row in soup.find_all('div', 'gs_ri'):
           
2. Line 137
            databox = __data    
            
```
comment
created
OrganicIrradiation
username_3
I believe this issue has been fixed with the current code. Could you please confirm?
comment
created
cozy9
username_0
The examples work now. Thanks!
comment
closed
OrganicIrradiation
username_3
issue