Summarization output different when using pipelines and the on-website inference engine.
Author: anchit-sadana
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0+cu113 (False)
- Tensorflow version (GPU?): 2.8.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False
```


### Who can help?

@Narsil 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See the colab notebook

https://colab.research.google.com/drive/1ALpWzqA0KMJ6K2SUhg8K-NAFhpFpoBm8?usp=sharing

The summary generated is completely different (and lower quality) from the one when I try to do a single inference on the website with the associated model at https://huggingface.co/sshleifer/distilbart-cnn-12-6. (With the text to be summarize at this [link](https://huggingface.co/sshleifer/distilbart-cnn-12-6?text=Andy+Murray+came+close+to+giving+himself+some+extra+preparation+time+for+his+wedding+next+week+before+ensuring+that+he+still+has+unfinished+tennis+business+to+attend+to.%0A%0AThe+world+No+4+is+into+the+semi-finals+of+the+Miami+Open%2C+but+not+before+getting+a+scare+from+21+year-old+Austrian+Dominic+Thiem%2C+who+pushed+him+to+4-4+in+the+second+set+before+going+down+3-6+6-4%2C+6-1+in+an+hour+and+three+quarters.+Murray+was+awaiting+the+winner+from+the+last+eight+match+between+Tomas+Berdych+and+Argentina%27s+Juan+Monaco.%0A%0APrior+to+this+tournament+Thiem+lost+in+the+second+round+of+a+Challenger+event+to+soon-to-be+new+Brit+Aljaz+Bedene.%0A%0AAnd+Murray+has+a+fairly+simple+message+for+any+of+his+fellow+British+tennis+players+who+might+be+agitated+about+his+imminent+arrival+into+the+home+ranks%3A+don%27t+complain.%0A%0AInstead+the+British+No+1+believes+his+colleagues+should+use+the+assimilation+of+the+world+number+83%2C+originally+from+Slovenia%2C+as+motivation+to+better+themselves.%0A%0AAt+present+any+grumbles+are+happening+in+private%2C+and+Bedene%27s+present+ineligibility+for+the+Davis+Cup+team+has+made+it+less+of+an+issue%2C+although+that+could+change+if+his+appeal+to+play+is+allowed+by+the+International+Tennis+Federation.%0A%0AMurray+thinks+anyone+questioning+the+move%2C+now+it+has+become+official%2C+would+be+better+working+on+getting+their+ranking+closer+to+his.%0A%0A%27If+he+was+500+in+the+world+they+wouldn%27t+be+that+fussed+about+it+but+obviously+he+threatens+their+position+a+bit%2C%27+said+the+27+year-old+Scot.+%27+and+he%27s+obviously+the+British+number+two%2C+comfortably.%0A%0A%27So+they+can+complain+but+the+best+thing+to+do+is+use+it+in+the+right+way+and+accept+it+for+what+it+is%2C+and+try+to+use+it+as+motivation+whether+they+agree+with+it+or+not.+He%27s+British+now+so+they%27ve+just+got+to+deal+with+it.%0A%0A%27I+would+hope+that+all+the+guys+who+are+below+him+now+like+James+%28Ward%29+%2C+Kyle+%28Edmund%29+%2C+Liam+%28Broady%29+they+will+use+it+as+motivation.+If+he+becomes+eligible+for+Davis+Cup+then+those+guys+are+going+to+have+to+prove+themselves.%0A%0A%27It+can+only+be+seen+as+a+positive+for+those+guys+using+it+to+try+to+get+better.+He%27s+a+good+player+but+so+are+James+and+Kyle+and+Liam+has+improved.+Aljaz+is+there%2C+he%27s+on+the+tour+every+week%2C+the+other+guys+aren%27t+quite+there+yet.%27%0A%0AFor+the+first+time+Murray%2C+who+has+an+encyclopaedic+knowledge+of+the+top+100%2C+gave+his+opinion+of+Bedene%3A+%27He%27s+a+good+player+with+a+very+good+serve.+He%27s+a+legitimate+top+100+player%2C+when+he+plays+Challengers+he%27s+there+or+thereabouts%2C+when+he+plays+on+the+main+tour+he+wins+matches%2C+it%27s+not+like+he+turns+up+and+always+loses+in+the+first+round.%0A%0A%27He+had+a+bad+injury+last+year+%28wrist%29+but+has+recovered+well.+I+would+imagine+he+would+keep+moving+up+the+rankings+although+I+don%27t+know+exactly+how+high+he+can+go.+I%27ve+practised+with+him+a+couple+of+times%2C+I+haven%27t+seen+him+play+loads%2C+but+when+you+serve+as+well+as+he+does+it+helps.+I+would+imagine+he%27+s+going+to+be+comfortably+in+the+top+70+or+80+in+the+world+for+a+while.%27%0A%0AIt+is+understood+the+Lawn+Tennis+Association+will+give+background+support+to+his+case+regarding+the+Davis+Cup+but+have+made+it+clear+that+the+onus+is+on+him+to+lead+the+way.+An+official+statement+said%3A+%27To+have+another+player+in+the+men%27s+top+100+is+clearly+a+positive+thing+for+British+tennis+and+so+we+very+much+welcome+Aljaz%27s+change+in+citizenship.%27%0A%0AThe+last+comparable+switch+came+twenty+years+ago+when+Greg+Rusedski+arrived+from+Canada.+It+was+by+no+means+universally+popular+but%2C+like+Bedene%2C+he+pledged+that+he+was+in+for+the+long+haul+and%2C+in+fairness+to+him%2C+he+proved+true+to+his+word.%0A%0AMurray+had+to+put+such+matters+aside+as+he+tackled+the+unusually+talented+Thiem%2C+a+delight+to+watch.+Coached+by+Boris+Becker%27s+veteran+mentor+Gunter+Bresnik%2C+he+slightly+resembles+Andy+Roddick+and+hits+with+similar+power+but+more+elegance.+His+single+handed+backhand+is+a+thing+of+rare+beauty.%0A%0AHowever%2C+he+has+had+a+mediocre+season+coming+into+this+event+and+there+was+little+to+forewarn+of+his+glorious+shotmaking+that+seemed+to+catch+Murray+unawares+early+on.%0A%0AThe+world+No+4+looked+to+have+worked+him+out+in+the+second%2C+but+then+suffered+one+of+his+periopdic+mental+lapses+and+let+him+back+in+from+4-1+before+closing+it+out+with+a+break.+After+breaking+him+for+3-1+in+the+decider+the+Austrian+whirlwind+burnt+itself+out.%0A%0A%27He%27s+a+strong+guy+who+hits+the+ball+hard+and+it+became+a+very+physical+match%2C%27+said+Murray.))


### Expected behavior

```shell
I would expect the summary generated from the pipeline to be identical to the one generated from the single inference on the website seen at the link above ```


Comments:
Hi @anchit-sadana ,

I couldn't load your colab (invalid credentials)

Summarization is not deterministic by default, so results are not always identical.

If you want identical results you can use `do_sample=False` parameter.
```python
pipe = pipeline(...)
out = pipe(text, do_sample=False)
```
or using the `{"parameters": {"do_sample": False}}` key within the API https://huggingface.co/docs/api-inference/detailed_parameters#summarization-task

You can also play with `min_length` and `max_length` to play with the results of the summarization.

Does that answer your question ?

Cheers,
Nicolas


Hi Nicolas,

Thanks for the response! I only posted this because I was consistently getting the same summary when I ran the pipeline object in colab and on the inference on the model card which were consistently different. Passing `do_sample = False` also does not seem to be changing the summary. Here's the fixed link to colab notebook - https://colab.research.google.com/drive/1ALpWzqA0KMJ6K2SUhg8K-NAFhpFpoBm8?usp=sharing. My apologies on the messing up the link earlier

Cheers, 
Anchit

Hi @anchit-sadana ,

I think the issue comes from the fact that your text is adding extra newlines between lines everywhere, so the model is not seeing the same text so summary is degraded.

(Probably didn't see a lot of double newlines during training and it's throwing the model a little off ?)


Ah that makes sense. Thanks for the response! Marking the issue as closed