Repository: lycheeverse/lychee

Issue #918: `Collector::skip_missing_input` is confusing
Created: 2023-01-16T21:19:17Z
The field skip_missing_input of crate::Collector is confusing:

  
    
      lychee/lychee-lib/src/collector.rs
    
    
        Lines 13 to 18
      in
      e240608
    
  
  
    

        
          
           pub struct Collector { 
        

        
          
               base: Option<Base>, 
        

        
          
               skip_missing_inputs: bool, 
        

        
          
               include_verbatim: bool, 
        

        
          
               use_html5ever: bool, 
        

        
          
           } 
        
    
  


It's used in Collector::collect_links()

  
    
      lychee/lychee-lib/src/collector.rs
    
    
        Lines 60 to 66
      in
      e240608
    
  
  
    

        
          
               pub async fn collect_links(self, inputs: Vec<Input>) -> impl Stream<Item = Result<Request>> { 
        

        
          
                   let skip_missing_inputs = self.skip_missing_inputs; 
        

        
          
                   let contents = stream::iter(inputs) 
        

        
          
                       .par_then_unordered(None, move |input| async move { 
        

        
          
                           input.get_contents(skip_missing_inputs).await 
        

        
          
                       }) 
        

        
          
                       .flatten(); 
        
    
  


which is passed to crate::Inputs::get_contents():

  
    
      lychee/lychee-lib/src/types/input.rs
    
    
        Lines 179 to 182
      in
      e240608
    
  
  
    

        
          
               pub async fn get_contents( 
        

        
          
                   self, 
        

        
          
                   skip_missing: bool, 
        

        
          
               ) -> impl Stream<Item = Result<InputContent>> { 
        
    
  



The first site where this argument is used is:

  
    
      lychee/lychee-lib/src/types/input.rs
    
    
        Lines 185 to 192
      in
      e240608
    
  
  
    

        
          
           InputSource::RemoteUrl(ref url) => { 
        

        
          
               let content = Self::url_contents(url).await; 
        

        
          
               match content { 
        

        
          
                   Err(_) if skip_missing => (), 
        

        
          
                   Err(e) => Err(e)?, 
        

        
          
                   Ok(content) => yield content, 
        

        
          
               } 
        

        
          
           }, 
        
    
  


where Input::url_contents() may return one of the following errors:

  
    
      lychee/lychee-lib/src/types/input.rs
    
    
        Lines 270 to 277
      in
      e240608
    
  
  
    

        
          
           let res = reqwest::get(url.clone()) 
        

        
          
               .await 
        

        
          
               .map_err(ErrorKind::NetworkRequest)?; 
        

        
          
           let input_content = InputContent { 
        

        
          
               source: InputSource::RemoteUrl(Box::new(url.clone())), 
        

        
          
               file_type, 
        

        
          
               content: res.text().await.map_err(ErrorKind::ReadResponseBody)?, 
        

        
          
           }; 
        
    
  


If it comes from reqwest::get(), it is critical and we should not silently skip it because the same error is almost likely to hit us again and again:
https://github.com/seanmonstar/reqwest/blob/6f714f438a4978afc1811f7c599315681cc8f205/src/lib.rs#L254-L261
If it comes from reqwest::Response::text(), that means the HTTP request returns an empty response body. This certainly could be an error users are interested. So it's reasonable to have an option to treat it as an error or not. But the naming of such option could be more appropriate such as deny_empty_response_body (which defaults to false).

The second site is:

  
    
      lychee/lychee-lib/src/types/input.rs
    
    
        Lines 238 to 248
      in
      e240608
    
  
  
    

        
          
           } else { 
        

        
          
               if self.is_excluded_path(path) { 
        

        
          
                   return (); 
        

        
          
               } 
        

        
          
               let content = Self::path_content(path).await; 
        

        
          
               match content { 
        

        
          
                   Err(_) if skip_missing => (), 
        

        
          
                   Err(e) => Err(e)?, 
        

        
          
                   Ok(content) => yield content, 
        

        
          
               }; 
        

        
          
           } 
        
    
  


where it returns an error if the input is a file and cannot be accessed:

  
    
      lychee/lychee-lib/src/types/input.rs
    
    
        Lines 333 to 335
      in
      e240608
    
  
  
    

        
          
           let content = tokio::fs::read_to_string(&path) 
        

        
          
               .await 
        

        
          
               .map_err(|e| ErrorKind::ReadFileInput(e, path.clone()))?; 
        
    
  


This certainly is an hard error that lychee should always report!

Duplicate issue #665: Inputs with no content to parse are considered valid
Created: 2022-06-30T04:32:47Z
With remote inputs, if the URI is to a web server and the page does not exist, it may respond with (or redirect to) a 404 page.

In some cases, you may want to use lychee to link check the content of a custom 404 page itself, but that should probably be opt-in?
In my case, the server would respond with 404 status, but no additional content (response header content-length: 0).

This was a broken input link, but lychee received a response, and presumably considered it valid, despite no content to parse?




Is this expected behaviour for lychee? When does it consider the input invalid?
Is it only a failure for inputs when:

The request times out
DNS lookup fails

A response with content-length: 0 might be valid for an actual link being checked (while the 404 status may not), but for inputs at least, no content seems like a bad input? (unless it is dynamic, like a queue)

It might be useful to log a warning about such inputs. Or possibly fail when no inputs have any content? The only hint of such presently would be that no links were found.

Closing comment (2025-01-06T17:36:31Z): Also, I just realized this is a partial duplicate of #918. Since there's a second issue, maybe we should look at it again. Let's move the discussion to the other issue as it's older.