Repository: alltheplaces/alltheplaces

Issue:
@spider=western_union in Poland list `amenity=money_transfer` POIs not actually existing as separate objects
very similar to #5881

It would be better to drop main tag over showing it like this. And in this case it seems dubious to me is it mappable as all on https://www.openstreetmap.org/node/5873034793 bank note.

https://www.alltheplaces.xyz/map/#16.47/50.076332/20.032325

https://location.westernunion.com/pl/malopolskie/krakow/e6d7165e8f86df94dacd8de6f1bfc780

I can visit that place and check in which form Western Union appears there.
[WesternUnion] Remove top level tag
Fixes #5889
@spider=western_union in Poland list `amenity=money_transfer` POIs not actually existing as separate objects
very similar to #5881

It would be better to drop main tag over showing it like this. And in this case it seems dubious to me is it mappable as all on https://www.openstreetmap.org/node/5873034793 bank note.

https://www.alltheplaces.xyz/map/#16.47/50.076332/20.032325

https://location.westernunion.com/pl/malopolskie/krakow/e6d7165e8f86df94dacd8de6f1bfc780

I can visit that place and check in which form Western Union appears there.


Discussion:
Looking further, basically all `amenity=money_transfer` in Poland are not really helpful in any way at all.

Or at least I cannot figure out how to use this data and it just obscures valid and useful data.

I think that some standalone `amenity=money_transfer` points may exist but vast vast majority of what is listed here is unrelated POI providing some service that seems to be not worth tagging in OSM.
Probably better as `money_transfer`, no top level tag. `money_transfer=western_union`? `money_transfer=yes`?
I know that ATP tags have no direct match but for reference: https://taginfo.openstreetmap.org/keys/money_transfer#overview

`money_transfer=western_union` seems OK, with caveat that also this is not viable for import (as place may support many money transfer networks)
I guess that https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/western_union.py#L15 should be modified, not figured out yet how to set arbitrary tags or how to add new category (`"extras": Categories.MONEY_TRANSFER.value` likely needs to be changed).

Looked at https://github.com/alltheplaces/alltheplaces/blob/master/README.md and found no obvious hints.
So maybe fix would be to modify 

https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/western_union.py#L15C105-L15C105 

to new value `WESTER_UNION_MONEY_TRANSFER_AS_ADDITIONAL_SERVICE` (I do not really like this name) and add at

https://github.com/alltheplaces/alltheplaces/blob/master/locations/categories.py#L157

`    WESTER_UNION_MONEY_TRANSFER_AS_ADDITIONAL_SERVICE = {"money_transfer": "western_union"}`

?

Or maybe https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/western_union.py#L15 should get `{"money_transfer": "western_union"}` directly?

(I looked at https://github.com/alltheplaces/alltheplaces#contributing-code but I see no obvious way to run specific spider to test is it still working - ideally at smaller scale without pulling all items)

Or maybe add a new field "`may_be_service_offered_within_poi_rather_standalone_poi=yes`" to properties of this spider?
> Or maybe https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/western_union.py#L15 should get `{"money_transfer": "western_union"}` directly?

This would be my vote.

> (I looked at https://github.com/alltheplaces/alltheplaces#contributing-code but I see no obvious way to run specific spider to test is it still working - ideally at smaller scale without pulling all items)

There are some [scrapy](https://docs.scrapy.org/en/latest/topics/extensions.html?highlight=close#std-setting-CLOSESPIDER_ITEMCOUNT) options for closing spiders early, but I think they are built on requests, not items, so they behave a little weird.

`-s CLOSESPIDER_ITEMCOUNT=10` will give you a minimum of 10 pois. So:

`pipenv run scrapy crawl -s CLOSESPIDER_ITEMCOUNT=10 western_union -O western_union.geojson`

If you are testing for a while/lots of requests, I recommend using a cache.
`-s HTTPCACHE_ENABLED=True -s HTTPCACHE_EXPIRATION_SECS=604800`

> Or maybe add a new field "`may_be_service_offered_within_poi_rather_standalone_poi=yes`" to properties of this spider?

My hope would be that all real POIs get categories, then the ones without mean this, however explicitly tagging them may be only option in reality. I think we should revisit this after the current push for categories.

Looking further, basically all `amenity=money_transfer` in Poland are not really helpful in any way at all.

Or at least I cannot figure out how to use this data and it just obscures valid and useful data.

I think that some standalone `amenity=money_transfer` points may exist but vast vast majority of what is listed here is unrelated POI providing some service that seems to be not worth tagging in OSM.
Probably better as `money_transfer`, no top level tag. `money_transfer=western_union`? `money_transfer=yes`?
I know that ATP tags have no direct match but for reference: https://taginfo.openstreetmap.org/keys/money_transfer#overview

`money_transfer=western_union` seems OK, with caveat that also this is not viable for import (as place may support many money transfer networks)
I guess that https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/western_union.py#L15 should be modified, not figured out yet how to set arbitrary tags or how to add new category (`"extras": Categories.MONEY_TRANSFER.value` likely needs to be changed).

Looked at https://github.com/alltheplaces/alltheplaces/blob/master/README.md and found no obvious hints.
So maybe fix would be to modify 

https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/western_union.py#L15C105-L15C105 

to new value `WESTER_UNION_MONEY_TRANSFER_AS_ADDITIONAL_SERVICE` (I do not really like this name) and add at

https://github.com/alltheplaces/alltheplaces/blob/master/locations/categories.py#L157

`    WESTER_UNION_MONEY_TRANSFER_AS_ADDITIONAL_SERVICE = {"money_transfer": "western_union"}`

?

Or maybe https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/western_union.py#L15 should get `{"money_transfer": "western_union"}` directly?

(I looked at https://github.com/alltheplaces/alltheplaces#contributing-code but I see no obvious way to run specific spider to test is it still working - ideally at smaller scale without pulling all items)

Or maybe add a new field "`may_be_service_offered_within_poi_rather_standalone_poi=yes`" to properties of this spider?
> Or maybe https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/western_union.py#L15 should get `{"money_transfer": "western_union"}` directly?

This would be my vote.

> (I looked at https://github.com/alltheplaces/alltheplaces#contributing-code but I see no obvious way to run specific spider to test is it still working - ideally at smaller scale without pulling all items)

There are some [scrapy](https://docs.scrapy.org/en/latest/topics/extensions.html?highlight=close#std-setting-CLOSESPIDER_ITEMCOUNT) options for closing spiders early, but I think they are built on requests, not items, so they behave a little weird.

`-s CLOSESPIDER_ITEMCOUNT=10` will give you a minimum of 10 pois. So:

`pipenv run scrapy crawl -s CLOSESPIDER_ITEMCOUNT=10 western_union -O western_union.geojson`

If you are testing for a while/lots of requests, I recommend using a cache.
`-s HTTPCACHE_ENABLED=True -s HTTPCACHE_EXPIRATION_SECS=604800`

> Or maybe add a new field "`may_be_service_offered_within_poi_rather_standalone_poi=yes`" to properties of this spider?

My hope would be that all real POIs get categories, then the ones without mean this, however explicitly tagging them may be only option in reality. I think we should revisit this after the current push for categories.

Resolving patch:
diff --git a/locations/spiders/western_union.py b/locations/spiders/western_union.py
--- a/locations/spiders/western_union.py
+++ b/locations/spiders/western_union.py
@@ -4,7 +4,6 @@
 from scrapy.downloadermiddlewares.retry import get_retry_request
 from scrapy.http import JsonRequest
 
-from locations.categories import Categories
 from locations.dict_parser import DictParser
 from locations.geo import point_locations
 from locations.hours import OpeningHours
@@ -12,7 +11,11 @@
 
 class WesternUnionSpider(Spider):
     name = "western_union"
-    item_attributes = {"brand": "Western Union", "brand_wikidata": "Q861042", "extras": Categories.MONEY_TRANSFER.value}
+    item_attributes = {
+        "brand": "Western Union",
+        "brand_wikidata": "Q861042",
+        "extras": {"money_transfer": "western_union"},
+    }
     allowed_domains = ["www.westernunion.com"]
     # start_urls[0] is a GraphQL endpoint.
     start_urls = ["https://www.westernunion.com/router/"]


Test patch: