Repository: pyload/pyload

Issue:
Implemented StreamcloudEu plugin based on XFileSharingPro
Resolves #128



Discussion:
A couple of suggestions:
- If the DIRECT_LINK_PATTERN is correctly defined you shouldn't have to override getDownloadLink.
- I suggest you to use the super statement to modify the setup method.

The reason I have overridden the getDownloadLink() function is that pyLoad received the download page multiple times without the POST parameters.
This caused the getDownloadLink() function of XFileSharingPro to throw an exception (the form wasn't found).
I am not sure why pyLoad got the download page immediately, but I am going to investigate it further.

Well, that's good enough if it works. Tell me if you want to try to fix it or merge it as it is.

I would like to fix it, I hope to get it done tomorrow.
Sorry for accidentally closing the merge.

I found out that the default (hidden) form is bypassed by pretending to have deactivated an adblocker. Afterwards, whenever the page (of this one video) is requested with the same session-cookie the video is displayed directly.
That isn't a problem when the page is only loaded once, but there might be reloads of the page, e.g. after the downloaded has been interrupted

tl;dr: To ensure a download, the initial check for the DIRECT_LINK_PATTERN is necessary.

Good, so if you agree I will merge.

Yes, please merge

This plugin is broken:
http://forum.pyload.org/viewtopic.php?f=10&t=2738

Calling `super(XFileSharingPro, self).getDownloadLink()` is not valid, you should use `super(StreamcloudEu, self).getDownloadLink()` instead, but that's not enough to fix the issue, the 'Form not found' error appears.

Yes, I tried to fix it, but the problem is quite complicated to solve.
The problem is that Streamcloud doesn't show the video if a Referer header is set.
But I haven't found any way to erase the Referer from the pycurl Instance.
The solution would probably be to create a new HTTPRequest instance and pass it all necessary attributes from the main Browser object.

Set `ref=False` in load don't work?

I didn't understand the behavior of `ref=False`. When a `Browser` object is used and `ref=False`, the Referer isn't set, instead the Referer from the previous request is used.

I don't understand what you mean. You have to avoid the Referer header right? so ty to set the ref parameter of the load function to False.

I did, but the only effect of setting the ref parameter to false is, that no new referer is set:

```
class HTTPRequest(self, url, get, post, referer, cookies, multipart=False):
    ...
    def setRequestContext():
        ...
        if referer and self.lastURL:
            self.c.setopt(pycurl.REFERER, str(self.lastURL))
```

But because the same pycurl object is used (stored in self.c in the HTTPRequest), the referer from the previous request is used again. Also, `pycurl.REFERER` only accepts strings, I tried to set it to `0`, `False` and `None`.
When setting it to an empty string, the Referer header is set to an empty string instead of being omitted.

Update: I am now working with a temporary `HTTPRequest` object instead of using `self.load`. It works so far, but the download page still isn't shown. I am going to investigate this further.

Okay, I got it to work. Apparently Streamcloud is currently increasing their bot protection.
Additionally to the not-allowed Referer header it is now necessary to wait a few seconds before requesting the page with the embedded video.

However, there seem to be a problem with the storage servers, I get a "Wrong IP" error for every video I try to watch (might just be my internet connection).

You can open a new pull request when you have a fixed version.

I just opened one, is there a way to hide my merge commits (pyload/stable → seoester/stable) or a better way to update a fork without creating a pull request?

Don't update your fork if you don't need it. You could delete your fork, create a new one and commit the fixed plugin. If you want to proceed this way close #178 and open a new pull request with the new fork.

You could create a branch for every feature/fix you develop. Your stable branch would be unaffected and you could simply update it with the upstream changes without merging.
Then we can merge the changes from your branch. This is the most common way to develop new features and merge them into the main branch.

Thanks guys, I am going to delete my fork and from now on create a new branch for every feature.


Resolving patch:
diff --git a/module/plugins/hoster/StreamcloudEu.py b/module/plugins/hoster/StreamcloudEu.py
--- a/module/plugins/hoster/StreamcloudEu.py
+++ b/module/plugins/hoster/StreamcloudEu.py
@@ -1,5 +1,7 @@
 # -*- coding: utf-8 -*-
 from module.plugins.hoster.XFileSharingPro import XFileSharingPro, create_getInfo
+from module.network.HTTPRequest import HTTPRequest
+from time import sleep
 import re
 
 class StreamcloudEu(XFileSharingPro):
@@ -15,7 +17,7 @@ class StreamcloudEu(XFileSharingPro):
     DIRECT_LINK_PATTERN = r'file: "(http://(stor|cdn)\d+\.streamcloud.eu:?\d*/.*/video\.mp4)",'
 
     def setup(self):
-        super(XFileSharingPro, self).setup()
+        super(StreamcloudEu, self).setup()
         self.multiDL = True
 
     def getDownloadLink(self):
@@ -23,6 +25,87 @@ def getDownloadLink(self):
         if found:
             return found.group(1)
 
-        return super(XFileSharingPro, self).getDownloadLink()
+        for i in range(5):
+            self.logDebug("Getting download link: #%d" % i)
+            data = self.getPostParameters()
+            httpRequest = HTTPRequest(options=self.req.options)
+            httpRequest.cj = self.req.cj
+            sleep(10)
+            self.html = httpRequest.load(self.pyfile.url, post = data, referer=False, cookies=True, decode = True)
+            self.header = httpRequest.header
+
+            found = re.search("Location\s*:\s*(.*)", self.header, re.I)
+            if found:
+                break
+
+            found = re.search(self.DIRECT_LINK_PATTERN, self.html, re.S)
+            if found:
+                break
+
+        else:
+            if self.errmsg and 'captcha' in self.errmsg:
+                self.fail("No valid captcha code entered")
+            else:
+                self.fail("Download link not found")
+
+        return found.group(1)
+
+    def getPostParameters(self):
+        for i in range(3):
+            if not self.errmsg: self.checkErrors()
+
+            if hasattr(self,"FORM_PATTERN"):
+                action, inputs = self.parseHtmlForm(self.FORM_PATTERN)
+            else:
+                action, inputs = self.parseHtmlForm(input_names={"op": re.compile("^download")})
+
+            if not inputs:
+                action, inputs = self.parseHtmlForm('F1')
+                if not inputs:
+                    if self.errmsg:
+                        self.retry()
+                    else:
+                        self.parseError("Form not found")
+
+            self.logDebug(self.HOSTER_NAME, inputs)
+
+            if 'op' in inputs and inputs['op'] in ('download1', 'download2', 'download3'):
+                if "password" in inputs:
+                    if self.passwords:
+                        inputs['password'] = self.passwords.pop(0)
+                    else:
+                        self.fail("No or invalid passport")
+
+                if not self.premium:
+                    found = re.search(self.WAIT_PATTERN, self.html)
+                    if found:
+                        wait_time = int(found.group(1)) + 1
+                        self.setWait(wait_time, False)
+                    else:
+                        wait_time = 0
+
+                    self.captcha = self.handleCaptcha(inputs)
+
+                    if wait_time: self.wait()
+
+                self.errmsg = None
+                self.logDebug("getPostParameters {0}".format(i))
+                return inputs
+
+            else:
+                inputs['referer'] = self.pyfile.url
+
+                if self.premium:
+                    inputs['method_premium'] = "Premium Download"
+                    if 'method_free' in inputs: del inputs['method_free']
+                else:
+                    inputs['method_free'] = "Free Download"
+                    if 'method_premium' in inputs: del inputs['method_premium']
+
+                self.html = self.load(self.pyfile.url, post = inputs, ref = False)
+                self.errmsg = None
+
+        else: self.parseError('FORM: %s' % (inputs['op'] if 'op' in inputs else 'UNKNOWN'))
+
 
 getInfo = create_getInfo(StreamcloudEu)


Test patch: